M. Brazelle1, B. Corey1, J. Porterfield1, B. Lindeman1 1University Of Alabama at Birmingham,Department Of Surgery,Birmingham, Alabama, USA
Introduction: A subset of Entrustable Professional Activities (EPAs) have been developed for general surgery. Little evidence currently exists for their validity as a workplace-based assessment tool. In this study, we aimed to collect concurrent validity evidence for EPAs as a framework for assessment of general surgery residents compared to the current gold standard of ACGME milestones.
Methods: All general surgery residents at a tertiary care center were encouraged to complete EPA micro-assessments using a mobile app. EPA ratings were collected across 3 phases of care (pre-, intra-, and post-operative) for 5 clinical scenarios immediately following a clinical encounter using a numerical scale from 0-4. Milestone evaluations (scored 0-4) were performed bi-annually to assess competency advancement. Resident PGY level and case difficulty were also collected. Correlation between EPA and milestone scores was assessed with Spearman rank correlation, and multivariable regression was utilized to identify significant predictors of milestone ratings.
Results: From August 2018-December 2019, a total of 320 micro-assessments were collected from 34 residents (70%). EPA scores demonstrated developmental increases by PGY level (Figure 1). Operative phase EPA scores were significantly lower than non-operative phase scores (2.04 ± 0.99 vs 3.70 ± 0.68, p,0.001). EPA scores demonstrated strong correlation with non-operative milestone ratings PC-1, MK-1, and ICS-2 (>0.55 for all, p<0.001) and moderate correlation with operative milestone ratings PC-3 and MK-2 (=0.41, p<0.001; =0.31, p<0.001). When EPA scores differed significantly from respective non-operative milestone ratings, EPA scores were consistently higher. On multivariable regression, PGY level (0.36, p<0.001) operative phase (-1.69, p<0.001) , and case difficulty (0.11, p=0.04) were significant independent predictors of faculty entrustment ratings.
Conclusion: The correlation between EPA micro-assessment scores and respective milestone ratings establishes concurrent validity evidence for EPAs as a formative assessment in general surgery residents. Operative phases of care were granted consistently less entrustment than non-operative phases. Further study is needed to determine how use of EPAs can enhance entrustment decision-making to better equip general surgery residents for unsupervised practice upon graduation.
