Abstract
Objective To study and profile the digital assessment behaviors of surgical faculty and residents, and to build a classifier to predict assessment completion, enhancing formative feedback initiatives.
Background As competency-based paradigms are integrated into surgical training, developing digital education tools for measuring competency and providing rapid feedback is crucial. Simply making assessments available is inadequate and results in disappointingly low user participation. To optimize engagement and efficacy of these tools, user assessment behaviors need to be studied.
Methods User data was aggregated from a HIPAA-compliant electronic health record (EHR)-integrated medical education platform. Faculty and resident behaviors were analyzed with respect to factors, such as time, day, device type, automated reminders, and EHR integration. A graphical convolutional neural network (GCN) model was trained to predict faculty participation in completing assessments.
Results 10,729 assessments were completed by 254 attendings for 428 residents across 22 institutions, from 2022 to 2024. 86% of assessments were completed by faculty on weekdays, were significantly influenced by automated platform triggers and EHR integration, and distinct faculty behavior profiles contingent upon time to completion and comment length were established. Residents opened assessments at a median of 1.5 hours of faculty assessment completion, with 96% of assessments viewed by 24 hours. The GCN model successfully predicted faculty assessment completion with 93.5% accuracy and an area under the ROC curve of 0.97.
Conclusions Faculty assessment behaviors represent an actionable bottleneck which can be leveraged to optimize and tailor the design of digital education tools, to enhance formative feedback.
Competing Interest Statement
I have read the journal's policy and the authors of this manuscript have the following competing interests: OHSU and Dr. Thanawala have a financial interest in Firefly Lab, LLC, a company that may have a commercial interest in the results of this research and technology. This potential conflict of interest has been reviewed and managed by OHSU.
Funding Statement
The author(s) received no specific funding for this work.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Not Applicable
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This project was reviewed and approved by the OHSU IRB committee. All data was anonymized prior to analysis.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Not Applicable
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Not Applicable
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Not Applicable
Footnotes
Conflicts of Interest/Disclosure Statement: OHSU and Dr. Thanawala have a financial interest in Firefly Lab, LLC, a company that may have a commercial interest in the results of this research and technology. This potential conflict of interest has been reviewed and managed by OHSU.
Funding/Financial Support Statement: No funding support was used for this study.
Data access statement: Data used in this study is available upon request.
Ethics Statement: This project was reviewed and approved by the OHSU IRB committee. All data was anonymized prior to analysis.
Data Availability
The data is premised on learner and hospital-based data which has been completely de-identified. Data governance policies of the research group require knowledge the individuals and intention of how the data would be used before sharing the de-identified datasets used in this study
