Original ReportsCorrelation of Objective Assessment Data With General Surgery Resident In-Training Evaluation Reports and Operative Volumes☆
Introduction
Although competent general surgeons continue to be produced from some 230+ American training programs, recent insights give our group pause for serious reflection on accurately assessing surgical resident performance: (1) The implementation of duty hour regulations1, 2 has led to increased concern that lower case volumes will affect graduating residents’ confidence and technical skill in performing open and laparoscopic surgery.3, 4 (2) Faculty evaluations can be biased, superficial, or offer minimal feedback on true resident performance. (3) The ABSITE provides no metric to assess a trainees’ technical skillset and judgement. With the increased importance of evaluating ACGME competencies for trainees in surgical residencies, as well as the implementation of the General Surgery Milestones,5 new methods of assessment that objectively evaluate residents are needed.
A more comprehensive assessment method with clearly defined learning objectives, timely and constructive feedback, and transparent and readily available score determinants is needed by surgical training programs and residents in order to chart a trajectory of performance.6, 7 This new assessment method must fit budgetary constraints and should be done efficiently to minimize time away from clinical and academic experience for both residents and staff. The growing body of evidence suggesting surgical training through simulation results in improved operating room (OR) performance8, 9 and better patient care outcomes10 has made this educational effort especially important at our institution. We hypothesized that performance in an OSCE-type (objective structured clinical examination) competition (Surgical X-Games) will prove more useful in identifying areas of strength or weakness among general surgery (GS) residents than faculty evaluations, ABSITE scores, or operative case logs.
Section snippets
X-Games
Five 15-minute testing stations each consisting of one or more surgical skill or knowledge-based tasks (Table 1) were developed and used within our multidisciplinary simulation center on 4 consecutive Fridays for each postgraduate class (PGY 2-5). The X-Games were held during the summer of 2013 and the winter of 2014. Each PGY class is tested separately in 2 groups on a Friday afternoon: 5 residents are assessed within the first 75 minutes and the second set of 5 assessed in the next 75
Results
A total of 35 GS residents (9 PGY-2s, 10 PGY-3s, 9 PGY-4s, and 7 PGY-5s) participated in both our summer (2013) and winter (2014) Surgical X-Games. Tests differed for each class and included a number of events: PGY-2s completed 15 events, PGY-3s completed 14, PGY-4s finished 13, and PGY-5s completed 12. Resident scores varied greatly in each class for X-Games, ABSITE, and OR case numbers. Staff evaluations varied little (scores given as median/range (Table 4). Higher overall scores on the
Discussion
Our study identified several important issues of resident assessment: (1) staff evaluations of individual residents varied little, with scores bunching up above the mean, (2) X-Games and ITER data raise concern regarding the objectivity of surgical staff evaluations, (3) operative case volumes positively correlate to performance on X-Games, and (4) X-Games were able to objectively assess resident performance and identify trainees needing remediation in specific tasks.
The results of this study
References (22)
- et al.
Perceptions of graduating general surgery chief residents: are they confident in their training?
J Am Coll Surg
(2014) - et al.
Graduating general surgery resident operative confidence: perspective from a national survey
J Surg Res
(2014) - et al.
The general surgery milestones project
J Am Coll Surg
(2014) - et al.
Laparoscopic simulation training with proficiency targets improves practice and performance of novice surgeons
Am J Surg
(2010) - et al.
Laparoscopic training on bench models: better and more cost effective than operating room experience?
J Am Coll Surg
(2000) - et al.
Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room-a randomized controlled trial
Am J Surg
(2010) - et al.
Assessment of technical skills transfer from the bench training model to the human model
Am J Surg
(1999) - et al.
The effect of bench model fidelity on endourological skills: a randomized controlled study
J Urol
(2002) - et al.
Relationship between objective assessment of technical skills and subjective in-training evaluations in surgical residents
J Am Coll Surg
(2004) - et al.
The measured effect of delay in completing operative performance ratings on clarity and detail of ratings assigned
J Surg Educ
(2014)
Cited by (16)
COVID-19 Did Not Stop the Rising Tide: Trends in Case Volume Logged by Surgical Residents
2023, Journal of Surgical EducationAssessment of Chief Resident Practice Readiness in a Porcine Lab: A 4-Year Experience
2022, Journal of Surgical EducationCitation Excerpt :The ABSITE is used across U.S. general surgery training programs as a measure of residents’ medical knowledge.12 While ABSITE scores have proven useful in predicting pass rates on the ABS Qualifying Exam (QE) and successful matching into a surgical fellowship,13,14 it has been shown several times that there is no correlation with technical performance.15,16,17 Scott et al. demonstrated that ABSITE scores had no correlation with either simulated operative testing nor intraoperative assessments.17
How do Trainers and Trainees use Formative Workplace-based Assessments of Operative Competence to Deliver and Receive Structured Feedback?
2022, Journal of Surgical EducationCitation Excerpt :Trainees at the early stages of their training are unlikely to be proficient in all component tasks and procedures necessary for independent practice. Abdelsattar et al.’s (2018) study from the Mayo clinic describes the tendency to describe the majority of trainees as “above average” as “Minnesota-nice”.25 “Failure to fail” is a phenomenon seen across educational grades, roles and professions in health professions education.26
International Medical Graduates are Comparable to American Medical Graduates as General Surgery Interns
2021, Journal of Surgical ResearchCitation Excerpt :Although standardized tests such as the ABSITE assess a trainee's breadth of medical knowledge, surgery is a field that requires practice to attain skill and dexterity. Therefore, we institute OSCE-style multistation evaluations in the form of the SO to the PGY-1 and the corresponding Surgical X-Games to PGY 2-5 residents.28-32 Our IMG preliminary residents were found to have higher overall scores on these assessments when compared with the AMG categorical residents; while these results may at least in part be reflective of the emphasis placed on these scores in selecting prelims who we take on as categorical residents and for advocating for IMG prelims during their subsequent residency application, this performance reiterates that foreign-trained physicians are able to rise up to such challenges and are competent in terms of their technical skills.
Does variability among surgical skills diminish throughout surgical internship? Analysis of a 5-task surgical simulation assessment program starting Day 1
2020, Surgery (United States)Citation Excerpt :Successful completion of both curricula are now required before sitting for the General Surgery Qualifying Examination of the American Board of Surgery.6 Few programs of the SBT that assess a broader range of technical skills, however, have been described.7,8 The skills curriculum of the American College of Surgeons and the Association of Program Directors in Surgery is a widely implemented, simulation-based skills curriculum for surgical interns that addresses 3 phases of training—core skills, advanced procedures, and team-based skills.9
The Experiential Benefit of an Orthopedic Trauma Fellowship: An Analysis of ACGME Case Log Data From 2006 to 2017
2019, Journal of Surgical EducationCitation Excerpt :Notably, the average resident completing an orthopedic trauma fellowship would be expected to more than triple their experience with “Open Complex” reductions, as defined by the ACGME, during their fellowship year (Table 2). This information is important, given the large body of literature tying case volume to surgeon skill and patient outcomes across a wide spectrum of procedures.3-14 As might be expected, for technically demanding trauma procedures, including acetabular, distal femur, and calcaneal fracture open reduction internal fixation, patient outcomes have been correlated with surgical experience.7-9
- ☆
Presented at the Western Surgical Association Annual Meeting, November 8-11, 2014 in Indian Wells, CA.