S. R. Pavuluri Quamme1, H. L. Beasley1, N. A. Brys1, H. N. Ghousseini1, D. A. Wiegmann1, Y. Ma1, J. C. Dombrowski1, G. E. Leverson1, L. L. Frasier1, C. C. Greenberg1 1University Of Wisconsin,Madison, WI, USA
Introduction: Peer surgical coaching is a promising approach for continuing professional development (CPD). The ability of surgeons to learn and apply core coaching principles and techniques is critical to the overall success of this approach to CPD. To date, however, no validated tool exists for reliably assessing coaches’ performance of these activities, making it difficult to evaluate the quality of coaching sessions or to provide formative feedback to coaches regarding their performance. The purpose of this study, therefore, was to develop and initially evaluate a tool for assessing coaching performance during peer surgical coaching sessions.
Methods: A multi-disciplinary team with expertise in surgery, education, human factors engineering, cognitive psychology, executive coaching, qualitative research, and psychometrics used an iterative approach to develop the rubric followed by initial validation. Extensive background research was conducted of the existing literature in coaching principles and approaches to identify the critical domains of coaching that were applicable in surgery. Audio recordings and corresponding transcripts from actual surgical coaching sessions were reviewed and scored by 7 team members independently. The team met to achieve consensus and clarify wording of the rubric until no further changes were required. Once consensus was achieved, 3 team members scored 8 coaching sessions. Gwet’s weighted agreement coefficient (AC) was used to evaluate inter-rater reliability as an initial step in validation.
Results: We identified four domains that are necessary for effective surgical coach performance: 1) shares responsibility, contributes to equal exchange; 2) uses questions/prompts to guide coachee self-reflection/analysis; 3) provides constructive feedback and encouragement; 4) guides goal setting and action planning. A five point scoring scale was designed to measure each of these performance domains as well as overall effectiveness: 1) counter-productive, 2) neutral/ineffective, 3) developing, 4) proficient, 5) exemplary. Detailed descriptions were developed as anchors for the scoring scale. Overall inter-rater reliability was measured at 0.78 AC.
Conclusions: We developed a scoring rubric to evaluate the overall and domain-specific effectiveness of a surgical coach. Initial validation shows good to excellent inter-rater reliability. While further validation is required and ongoing, this rubric can be used to provide feedback for the training and development of surgical coaches and to evaluate fidelity to the coaching model, a critical requirement for research in this area.
