Elsevier

Journal of Surgical Education

Volume 75, Issue 6, November–December 2018, Pages 1430-1436
Journal of Surgical Education

Original Reports
Correlation of Objective Assessment Data With General Surgery Resident In-Training Evaluation Reports and Operative Volumes

https://doi.org/10.1016/j.jsurg.2018.04.016Get rights and content

Objective

Faculty evaluations, ABSITE scores, and operative case volumes often tell little about true resident performance. We developed an objective structured clinical examination called the Surgical X-Games (5 rooms, 15 minutes each, 12-15 tests total, different for each postgraduate [PGY] level). We hypothesized that performance in X-Games will prove more useful in identifying areas of strength or weakness among general surgery (GS) residents than faculty evaluations, ABSITE scores, or operative cases volumes.

Design

PGY 2 to 5 GS residents (n = 35) were tested in a semiannual X-Games assessment using multiple simulation tasks: laparoscopic skills, bowel anastomosis, CT/CXR analysis, chest tube placement, etc. over 1 academic year. Resident scores were compared to their ABSITE, in-training evaluation reports, and operating room case numbers.

Setting

Academic medical center.

Participants

PGY-2, 3, 4, and 5 GS residents at Mayo Clinic in Rochester, MN.

Results

Results varied greatly within each class except for staff evaluations: in-training evaluation reports medians for PGY-2s were 5.3 (range: 5.0-6.0), PGY-3s 5.9 (5.5-6.3), PGY-4s 5.6 (5.0-6.0), and PGY-5s were 6.1 (5.6-6.9). Although ABSITE and operating room case volumes fluctated greatly with each PGY class, only X-Games scores (median: PGY-2 = 82, PGY-3 = 61, PGY-4 = 76, and PGY-5 = 60) correlated positively (p < 0.05) with operative case volume and negatively (p < 0.05) with staff evaluations.

Conclusions

X-Games assessment generated wide differentiation of resident performance quickly, inexpensively, and objectively. Although “Minnesota-nice” surgical staff may feel all GS trainees are “above average,” objective assessment tells us otherwise.

Introduction

Although competent general surgeons continue to be produced from some 230+ American training programs, recent insights give our group pause for serious reflection on accurately assessing surgical resident performance: (1) The implementation of duty hour regulations1, 2 has led to increased concern that lower case volumes will affect graduating residents’ confidence and technical skill in performing open and laparoscopic surgery.3, 4 (2) Faculty evaluations can be biased, superficial, or offer minimal feedback on true resident performance. (3) The ABSITE provides no metric to assess a trainees’ technical skillset and judgement. With the increased importance of evaluating ACGME competencies for trainees in surgical residencies, as well as the implementation of the General Surgery Milestones,5 new methods of assessment that objectively evaluate residents are needed.

A more comprehensive assessment method with clearly defined learning objectives, timely and constructive feedback, and transparent and readily available score determinants is needed by surgical training programs and residents in order to chart a trajectory of performance.6, 7 This new assessment method must fit budgetary constraints and should be done efficiently to minimize time away from clinical and academic experience for both residents and staff. The growing body of evidence suggesting surgical training through simulation results in improved operating room (OR) performance8, 9 and better patient care outcomes10 has made this educational effort especially important at our institution. We hypothesized that performance in an OSCE-type (objective structured clinical examination) competition (Surgical X-Games) will prove more useful in identifying areas of strength or weakness among general surgery (GS) residents than faculty evaluations, ABSITE scores, or operative case logs.

Section snippets

X-Games

Five 15-minute testing stations each consisting of one or more surgical skill or knowledge-based tasks (Table 1) were developed and used within our multidisciplinary simulation center on 4 consecutive Fridays for each postgraduate class (PGY 2-5). The X-Games were held during the summer of 2013 and the winter of 2014. Each PGY class is tested separately in 2 groups on a Friday afternoon: 5 residents are assessed within the first 75 minutes and the second set of 5 assessed in the next 75

Results

A total of 35 GS residents (9 PGY-2s, 10 PGY-3s, 9 PGY-4s, and 7 PGY-5s) participated in both our summer (2013) and winter (2014) Surgical X-Games. Tests differed for each class and included a number of events: PGY-2s completed 15 events, PGY-3s completed 14, PGY-4s finished 13, and PGY-5s completed 12. Resident scores varied greatly in each class for X-Games, ABSITE, and OR case numbers. Staff evaluations varied little (scores given as median/range (Table 4). Higher overall scores on the

Discussion

Our study identified several important issues of resident assessment: (1) staff evaluations of individual residents varied little, with scores bunching up above the mean, (2) X-Games and ITER data raise concern regarding the objectivity of surgical staff evaluations, (3) operative case volumes positively correlate to performance on X-Games, and (4) X-Games were able to objectively assess resident performance and identify trainees needing remediation in specific tasks.

The results of this study

Cited by (16)

  • Assessment of Chief Resident Practice Readiness in a Porcine Lab: A 4-Year Experience

    2022, Journal of Surgical Education
    Citation Excerpt :

    The ABSITE is used across U.S. general surgery training programs as a measure of residents’ medical knowledge.12 While ABSITE scores have proven useful in predicting pass rates on the ABS Qualifying Exam (QE) and successful matching into a surgical fellowship,13,14 it has been shown several times that there is no correlation with technical performance.15,16,17 Scott et al. demonstrated that ABSITE scores had no correlation with either simulated operative testing nor intraoperative assessments.17

  • How do Trainers and Trainees use Formative Workplace-based Assessments of Operative Competence to Deliver and Receive Structured Feedback?

    2022, Journal of Surgical Education
    Citation Excerpt :

    Trainees at the early stages of their training are unlikely to be proficient in all component tasks and procedures necessary for independent practice. Abdelsattar et al.’s (2018) study from the Mayo clinic describes the tendency to describe the majority of trainees as “above average” as “Minnesota-nice”.25 “Failure to fail” is a phenomenon seen across educational grades, roles and professions in health professions education.26

  • International Medical Graduates are Comparable to American Medical Graduates as General Surgery Interns

    2021, Journal of Surgical Research
    Citation Excerpt :

    Although standardized tests such as the ABSITE assess a trainee's breadth of medical knowledge, surgery is a field that requires practice to attain skill and dexterity. Therefore, we institute OSCE-style multistation evaluations in the form of the SO to the PGY-1 and the corresponding Surgical X-Games to PGY 2-5 residents.28-32 Our IMG preliminary residents were found to have higher overall scores on these assessments when compared with the AMG categorical residents; while these results may at least in part be reflective of the emphasis placed on these scores in selecting prelims who we take on as categorical residents and for advocating for IMG prelims during their subsequent residency application, this performance reiterates that foreign-trained physicians are able to rise up to such challenges and are competent in terms of their technical skills.

  • Does variability among surgical skills diminish throughout surgical internship? Analysis of a 5-task surgical simulation assessment program starting Day 1

    2020, Surgery (United States)
    Citation Excerpt :

    Successful completion of both curricula are now required before sitting for the General Surgery Qualifying Examination of the American Board of Surgery.6 Few programs of the SBT that assess a broader range of technical skills, however, have been described.7,8 The skills curriculum of the American College of Surgeons and the Association of Program Directors in Surgery is a widely implemented, simulation-based skills curriculum for surgical interns that addresses 3 phases of training—core skills, advanced procedures, and team-based skills.9

  • The Experiential Benefit of an Orthopedic Trauma Fellowship: An Analysis of ACGME Case Log Data From 2006 to 2017

    2019, Journal of Surgical Education
    Citation Excerpt :

    Notably, the average resident completing an orthopedic trauma fellowship would be expected to more than triple their experience with “Open Complex” reductions, as defined by the ACGME, during their fellowship year (Table 2). This information is important, given the large body of literature tying case volume to surgeon skill and patient outcomes across a wide spectrum of procedures.3-14 As might be expected, for technically demanding trauma procedures, including acetabular, distal femur, and calcaneal fracture open reduction internal fixation, patient outcomes have been correlated with surgical experience.7-9

View all citing articles on Scopus

Presented at the Western Surgical Association Annual Meeting, November 8-11, 2014 in Indian Wells, CA.

View full text