Comparison of simulation-based assessments and faculty ratings for general surgery resident milestone evaluation: Are they telling the same story?

https://doi.org/10.1016/j.amjsurg.2016.07.025Get rights and content

Highlights

  • We compared simulation assessments to traditional end-of-rotation evaluations.

  • Faculty could not rate many nontechnical competencies from clinical observations.

  • Simulation can inform resident nontechnical competency milestone evaluations.

Abstract

Background

The goal of this article is to examine the role of simulation in documenting resident nontechnical competencies outlined by the general surgery milestones, and compare those assessments to faculty ratings.

Methods

Trainees completed a multiphasic simulation scenario to assess ability to obtain informed consent, lead a preoperative time out, crisis management, communication, and delivering bad news. Assessments from this scenario were compared with ratings of these same competencies collected from clinical rotations.

Results

Twenty-six PGY1 trainees participated in the training program. Results revealed no significant correlations between simulation performance and faculty rotation ratings for any of the 5 competencies. With the exception of communication in the operating room, faculty reported an overall inability to reliably observe these competencies 26% to 41% of the time.

Conclusions

This work suggests that traditional end-of-rotation evaluations may not be the most feasible and valid method to evaluate resident nontechnical skills for milestone assessment. Simulation-based assessments should be considered to fill this gap.

Section snippets

Methods

Participants consisted of PGY1 trainees in the UT Southwestern General Surgery Residency program. The IRB deemed this project exempt. All training sessions took place in the last 2 months of Internship. Participants received pre training materials 1 week before attending the simulation session. Pretraining materials consisted of the SPIKES Delivering Bad News protocol,5 a surgical safety checklist from the World Health Organization (WHO),6 the Society of American Gastroenterologists and

Results

Twenty-six PGY1 residents participated in the simulation training.

Comments

This study was designed to examine the extent to which simulation can fulfill specific aspects of the milestone project's goals for resident competency. Our work suggests that simulated environments can be used to create a standardized avenue for assessing resident competency in these specific areas. In addition, these results suggest that simulation may provide unique data that is otherwise unobtainable from clinical assessments. Our study demonstrates that faculty may not have enough

Conclusions

Given the level of variability of competence among graduating general surgery residents,17, 18, 19, 20 methods to assess resident competence early and often and in a standardized fashion should be thoroughly developed and implemented across training programs. One of the goals of the milestones project was to direct and motivate programs toward this end. However, there is still confusion among programs regarding the best way to implement these milestones assessments,21, 22 as well as which

References (22)

  • S. Undre et al.

    Multidisciplinary crisis simulations: the way forward for training surgical teams

    World J Surg

    (2007)
  • Cited by (9)

    • Junior doctor experiences and challenges in obtaining surgical informed consent: A qualitative systematic review & meta-ethnography

      2021, Journal of Surgical Research
      Citation Excerpt :

      Few junior doctors have had the opportunity to acquire SIC under the direct supervision of a more senior doctor, with 73.3% of the interns surveyed by Heaney et al reporting they had never had such an opportunity.29 Thirteen of the thirty-four included studies recommend that additional educational interventions are implemented for all staff, including junior doctors, to improve skills and knowledge around SIC.8,19,27-31,36,41,43-46 Suggested interventions include assessment of junior doctors’ consenting skills in simulated environments22,42,46, bioethics and communication for junior doctors23,37, dedicated education programs for junior doctors to acquire SIC22; spending more time with senior doctors observing juniors’ skills in consenting, allowing juniors to have more theatre time to observe procedures29,31, ensuring surgical trainees have adequate understanding of the details of surgical procedures37, direct observation by senior doctors while acquiring SIC44,23,31,48, use of an online learning module44, use of required reading44, and ensuring consent is taught adequately in the medical school curriculum[31], ensuring trainees have access to further general didactic education on SIC.23,28,30,41

    • International Medical Graduates are Comparable to American Medical Graduates as General Surgery Interns

      2021, Journal of Surgical Research
      Citation Excerpt :

      Although several programs have developed innovative ways to assess their trainees, there is a paucity of standardized assessments that accurately map a resident's trajectory in terms of surgical skill. Gardner et al.33 found no significant correlation between simulation-based assessments and end-of-rotation faculty evaluations and identified that the latter could not accurately distinguish performance toward the set clinical competencies. In another article from the same institution, the authors proposed a comprehensive system that combined simulation, ABSITE scores, and subjective evaluations among others to critically analyze resident progress.34

    • What do quantitative ratings and qualitative comments tell us about general surgery residents’ progress toward independent practice? Evidence from a 5-year longitudinal cohort

      2019, American Journal of Surgery
      Citation Excerpt :

      Prior studies have noted concerns over rotation evaluation scores as not being reliable assessments of learner performance, despite their prevalent use in making promotion decisions.8–13 However, recent studies have shown that rotation evaluations scores can be reliable and demonstrate validity evidence, provided that evaluations from multiple raters are collected and aggregated over a sufficient period of time.7,14–21 This study focuses on synthesizing both quantitative and qualitative data from rotation evaluations, in the context of identifying competency-based developmental levels.

    • Emotional Intelligence and Delivering Bad News: The Jury is Still Out

      2019, Journal of Surgical Education
      Citation Excerpt :

      All phases of the training session were video recorded for performance assessment. Performance for DBN was evaluated using a 10-item checklist tool (achieved/did not achieve) internally developed based on best practice suggestions by Baile etal.,19 and used elsewhere.24 The checklist for DBN was completed at the completion of all simulation sessions via video review by 2 simulation faculty members (surgeon and PhD).

    View all citing articles on Scopus

    There were no relevant financial relationships or any sources of support in the form of grants, equipment, or drugs.

    The authors report no conflicts of interest.

    View full text