D. J. Cassidy1, S. Chakraborty2, N. Panda1, S. K. McKinley1, A. Mansur1, I. Hamdi1, J. T. Mullen1, E. Petrusa1, R. Phitayakorn1, D. Gee1 1Massachusetts General Hospital,Department Of Surgery,Boston, MA, USA 2Memorial Sloan-Kettering Cancer Center,Department Of Epidemiology And Biostatistics,New York, NY, USA
Introduction:
Resident performance on the American Board of Surgery In-Training Examination (ABSITE) is used for evaluation of surgical knowledge and guides resident selection for institutional remediation programs. Remediation thresholds have historically been based on ABSITE percentile scores; however, this does not account for predictors that can impact a resident’s exam performance. We sought to identify predictors of yearly ABSITE performance to help identify residents “at-risk” for falling off their knowledge growth trajectory.
Methods:
Six years (2014-2019) of resident score reports were deidentified and analyzed. The knowledge of the residents, as measured by their standard scores on the ABSITE, was modeled as a function of the corresponding postgraduate year via a linear mixed effect regression model. Time-invariant covariates analyzed included written USMLE1-3 examination scores. Yearly time-variant covariates analyzed included number of practice questions completed, percentage correct of practice questions (TrueLearn, LLC), and the resident’s post-graduate year, accounting for profession development sabbatical. For each resident, the predicted ABSITE standard score along with a 95% bootstrap prediction interval was obtained. The score of an individual was determined to have “fallen off the curve”, if the observed ABSITE score lied below the associated prediction interval.
Results:
A total of 376 score reports from 130 categorical general surgery residents were analyzed. Time-invariant covariates that had a significant effect on the model included USMLE1 (p<0.05 for PGY1-4) and USMLE2 scores (p<0.05 for PGY1-5), whereas USMLE 3 scores were not a significant predictor of ABSITE scores. Time-variant covariates that had a significant effect were number of practice questions completed (p=0.001), whereas percentage of practice questions correct (p=0.48) and number of years in the program (p=0.42) were not significant predictors. Five residents were identified as having “fallen off” their predicted knowledge curve, including a single resident on two occasions. Population prediction curves were obtained at 7 different covariate percentile levels (5%, 10%, 25%, 50%, 75%, 90%, and 95%) and plotted to predict resident knowledge progress [Figure 1].
Conclusion:
Performance on USMLE 1 and 2 examinations and number of practice questions completed continue to be a positive predictor of ABSITE performance. Creating residency-wide knowledge growth curves as well as individualized predictive ABSITE performance models allows for more efficient identification of residents potentially at risk for poor ABSITE performance and structured monitoring of surgical knowledge progression.
