Assessment of surgical training
Introduction
The training paradigm whereby trainees learn in the operating room (OR) through graded responsibility was introduced by William Halsted over a century ago.1 Today, reduced hours in training, financial constraints, and a focus on patient safety require a more structured training experience, as well as an assessment of surgical skills that documents clinical and operative proficiency, prior to completion of training. Evaluation and/or assessment can be objective or subjective. Objective evaluation is quantitative, norm referenced, and usually in the form of a single MCQ test; subjective evaluation often employs several observed parameters to develop a qualitative judgment.2 There is a tendency for these more subjective evaluations to cluster around the mean, particularly with longer times between observation and documentation.2, 3 Finally most evaluations performed during surgical training are subjective, are completed at the end of a rotation and often consist of a single number on a Likert scale or brief narrative that rarely permits charting a performance trajectory.2
Single institution studies suggest 20%–25% trainees have performance deficits.4, 5 In the USA the cumulative risks of termination are 3.0% and 19.5%, respectively, for surgical trainees.6 Although many “voluntary” resignations may not be entirely voluntary, there is still a discrepancy between the 3% who are terminated and the reported prevalence of problem trainees. While some might be successfully remediated; it is likely that many will graduate without correction, particularly when the deficiency is in interpersonal communication or professionalism.7 In one study deficiencies persisted at graduation in 88% of the trainees identified to have performance problems.5 Furthermore 82% of the behavior problems were identified in the first year of training.5 Finally although cause and effect between error and trainee deficiencies remains unclear, the impact of surgical error on patient outcomes is now apparent.8, 9, 10, 11
We have an obligation to ensure that graduating surgical trainees are technically proficient and have the non – technical skills necessary for safe, independent practice. It is not acceptable to assume that this learning will take place in practice or even during fellowship training. Frequently less attention is paid to evaluating and remediating the more critical non – technical skills than to ensuring technical proficiency.12 Therefore trainees are given a vague label of “can't operate” with non – specific recommendations for improvement such as “needs more experience”. The common intervention of providing stricter supervision (“watch how I do this one”) is likely to address only a minority of technical errors and delay recognition of a critical deficiency until the resident is in practice.9, 12 This paper will review the literature on the assessment of technical and non – technical operative skills and professionalism with recommendations for a more rigorous and meaningful evaluation.
Technical skills are defined as the ability to use surgical instruments in an effective and efficient manner.13 The assessment of these basic skills is critical to identify and correct technical deficiencies in junior trainees before these become ingrained, and to allow a focus on more complex patient care and management issues in the OR. Simulation skill training has been demonstrated to improve performance in the OR14, 15, 16, 17, 18, 19, 20, 21, 22 and accommodate differences in trainee individual trainee performance thereby ensuring that all can achieve proficiency.23 Furthermore this training approach results in graduating trainees who perform at a uniform level of proficiency24, 25 and who have fewer errors in the OR.26, 27 To date, the evidence for transfer to the OR is stronger for minimally invasive surgery than for open procedures.19, 28, 29, 30
Technical skills can be learned, assessed and remediated in the skills lab using a variety of validated instruments including the Objective Structured Assessment of Technical Skills (OSATS)31, 32, 33 from Toronto, the Verification of Proficiency system (VOP) from Southern Illinois University (SIU),34 the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS)28, 35 and the Imperial College Surgical Assessment Device (ICSAD).36, 37 The MISTELS performance metrics have been extensively validated with a passing score defined to differentiate competent from non-competent laparoscopic surgeons38 and have been incorporated as the manual skills component of the Fundamentals of Laparoscopic Surgery program.39 VOP is based on OSATS, but because performances are recorded on a remote video system direct faculty observation is unnecessary at the time of testing.34 Trainees who fail to demonstrate proficiency undergo a period of mandatory remediation with a faculty coach. To date all that failed at the first attempt subsequently achieved proficiency and passed on retesting.34
Available instruments for evaluation of more advanced technical skills in the OR include the Global Operative Assessment of Laparoscopic Skills (GOALS)40, 41 and the Operative Performance Rating System (OPRS).3, 42, 43, 44, 45, 46, 47 The GOALS instrument is a reliable and valid measure of technical skill when used to evaluate laparoscopic cholecystectomy, or laparoscopic appendectomy.40 The OPRS instruments were developed based on evidence-based parameters of good and poor patient outcomes and have been used to evaluate trainee performance for over a decade3, 42, 43, 44, 45, 46, 47
Operative performance evaluation is most effective in cases where the trainee does the majority of the operation and is senior enough to set up the case and direct the surgical team. Four categories of non – technical skill deficiency have been defined: deficiencies in Forward Planning, Self-Direction, Situation Awareness, and Patient Safety (including judgment).45 Forward planning is defined as the ability to anticipate needs, to think ahead and to set up the operative field in an optimum fashion. Self-Direction refers to the demeanor of the trainee and includes an ability to conduct oneself in a professional manner, stay focused, not become distracted, and slow down when appropriate. It also includes the ability to accept and respond to feedback. Expert surgeons are able to recognize when it is necessary to “slow down”, and engage in more effortful, analytic behavior.48 This conversion from routine technical behavior to an analytic, reflective behavior is modeled by slowing or stopping the progress of an operation when a critical point is encountered, until this behavior becomes automatic to the trainee.48 Factors inhibiting this conversion include fatigue, overconfidence, difficulties in team management and communication, distractions and time constraints.12, 48, 49
Assessment of a trainee's ability to set up the operation, maintain forward progress and slow down appropriately can be performed by direct observation in the OR by the consultant. Trainees should be encouraged to think each step aloud (or take a junior through the case) to display understanding of the critical “slowing down” steps, allowing the distinction to be made between a lack of confidence and a knowledge deficit.12 Probing residents with a series of “what if” or “what next” questions (Table 1) will not only elicit knowledge deficits but also promote improved future forward planning. Trainees with deficiencies in forward planning and/or self-direction could be assisted by participating in a preoperative mental rehearsal discussion of the critical operative steps with the consultant50, 51, 52, 53, 54 i.e. developing a “flight plan” of critical steps.12
Situation Awareness is the ability to assess and interpret cues from the environment and to provide team leadership. Acquiring these skills is challenging for trainees who are usually completely focused on the operative field.12 A number of checklist tools are available to teach and evaluate Situation Awareness. These include STEPP (Status of the patient, Team, Environment, and Progress towards the goal) (Table 2),55 SMART56 and NOTECHS.57 SMART (Situation, Management, Activity, Rapidity, Troubleshoot, and Talk to me) was developed for use with ad hoc trauma teams but could be adapted to any surgical crisis where urgent communication is essential.56 Sevdalis et al.57 adapted the NOTECHS instrument from aviation to evaluate leadership skills, situation awareness, decision making and communication in a simulated surgical environment with good reliability. Leadership skills are often best assessed by direct observation from health care professionals who have the opportunity to observe the trainee without consultant presence.
The category of Judgment and Patient Safety was defined as the ability to recognize and solve problems, and to avoid and recover from errors and unexpected events. These skills represent one difference between skilled, experienced workers and advanced experts.58 Good judgment is a function not only of experience but also of surgical knowledge, for which a number of standardized tests are available. One tool has been have described to assess cognitive skills and organizational ability that is completed online jointly by the trainee and the consultant at the end of each procedure.59 Since experts rely on automated knowledge to make critical operative decisions errors are made 50% of the time when attempting to describe a procedure. Cognitive Task Analysis (CTA) is one method to deconstruct these automated skills by creating a checklist of specific operative steps and decision points and has been used to enhance development of intra-operative decision making.60, 61, 62, 63, 64 While time consuming to develop, evidence-based CTA frameworks could not only improve instruction with simulated scenarios but could also allow for more accurate assessment of a trainee's performance and remediation needs.
With the current focus upon patient safety, the OR is not an ideal environment for remediation to occur. Simulation provides a viable and valid alternative for delivering educational training and remedial curricula and allows trainees to acquire skills that transfer to the OR in a safe, realistic setting without risk to the patient.27, 30, 65 Simulated environments also allow the trainee more independence and responsibility during the performance, thus allowing non – technical deficiencies to be observed and diagnosed. In addition more challenging scenarios can be introduced as the trainee's skill level improves.57, 66, 67, 68 The addition of Web-based clinical training sites that permit technical and cognitive rehearsal of procedures is likely to further enhance the transfer of judgmental expertise in the simulated environment.69
Section snippets
Professionalism
An emerging wealth of data has demonstrated that unprofessional behaviors contribute to medical errors, poor patient satisfaction, adverse outcomes,70 and an increase in the cost of care.71, 72 Such behaviors lower morale to the extent that other health care professionals seek new positions in more professional environments.73, 74 Furthermore a failure to address these issues impacts the morale of trainee colleagues who are expected to cover for the lapses of others. Therefore unprofessional
Discussion
The patient safety imperative78, 79 has raised expectations regarding the responsibility of medical educators and to decision makers to ensure that trainees are ready for independent practice upon graduation. The ability to accomplish this is challenged by reduced work hours such that trainees spend less time in the OR80 and perform fewer cases than desirable.81 Almost certainly the increasing number of trainees, who are choosing to do post-residency fellowship training, reflects their concern
Conclusions
Trainee performance should be evaluated in a rigorous, reliable and meaningful way to ensure that graduates have the skills necessary for safe, independent practice. It is not acceptable to assume that this learning will take place in practice or during fellowship training. While the assessment of basic surgical skills for junior trainees is critical to identify and correct technical deficiencies before these become ingrained, the assessment of non – technical skills is even more important for
References (95)
- et al.
The general surgery milestones project
JACS
(2014) - et al.
The nature of general surgery resident performance problems
Surgery
(2009) - et al.
Remediation practices for surgery residents
Am J Surg
(2009) - et al.
Analysis of errors reported by surgeons at three teaching hospitals
Surgery
(2003) Why Johnny can't operate
Surgery
(2009)- et al.
Laparoscopic simulation training with proficiency targets improves practice and performance of novice surgeons
AJS
(2010) - et al.
Simulator training for laparoscopic suturing using performance goals translates to the operating room
JACS
(2005) - et al.
Laparoscopic training on bench models: better and more cost effective than operating room experience?
JACS
(2000) - et al.
Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room – a randomized controlled trial
Am J Surg
(2010) - et al.
Redefining simulator proficiency using automaticity theory
Am J Surg
(2007)
Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies
Am J Surg
Developing criteria for proficiency-based training of surgical technical skills using simulation: changes in performances as a function of training year
JACS
Testing technical skills via an innovative “bench station” examination
Am J Surg
The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model
JACS
Evaluation of resident laparoscopic performance using global operative assessment of laparoscopic skills
JACS
A global assessment tool for evaluation of intraoperative laparoscopic skills
Am J Surg
Refining the evaluation of operating room performance
J Surg Educ
Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents
Surgery
Evaluating resident operative performance: a qualitative analysis of expert opinions
Surgery
A template for reliable assessment of resident operative performance: assessment intervals, numbers of observations and raters
Surgery
Comparing the effects of physical practice and mental imagery rehearsal on learning basic surgical skills by medical students
Am J Obstet Gynecol
The impact of brief team communication, leadership and team behavior training on ad hoc team performance in trauma care settings
Am J Surg
Reliability of a revised NOTECHS scale for use in surgical teams
Am J Surg
A new web-based operative skills assessment tool effectively tracks progression in surgical resident performance
J Surg Educ
Judgment analysis: a method for quantitative evaluation of trainee surgeons' judgments of surgical risk
Am J Surg
Thoracic surgery skill proficiency with chest wall tumor simulator
J Surg Res
Faculty evaluation of simulation-based modules for assessment of intraoperative decision making
Surgery
Outcome measures for surgical simulators: is the focus on technical skills the best approach?
Surgery
The use of cognitive task analysis to improve the learning of percutaneous tracheostomy placement
Am J Surg
Operating room teamwork among physicians and nurses: teamwork in the eye of the beholder
JACS
Cognitive task analysis for training
Int J Educ Res
The relation of patient satisfaction with complaints against physicians, risk management episodes, and malpractice lawsuits
Am J Med
How do supervising surgeons evaluate guidance provided in the operating room?
Am J Surg
The training of the surgeon
Bull Johns Hopkins Hosp
The measured impact of delay in completing operative performance ratings on clarity and detail of ratings assigned
J Surg Educ
Factors related to attrition in surgery residency based on application data
Arch Surg
A national study of attrition in general surgery training: which residents leave and where do they go?
Ann Surg
Human error, not communication and systems, underlies surgical complications
Surgery
Patterns of technical error among surgical malpractice claims: an analysis of strategies to prevent injury to surgical patients
Ann Surg
Hospital ownership and preventable adverse events
J Gen Intern Med
Recognizing residents with a deficiency in operative performance as a step closer to effective remediation
JACS
Toward feasible, valid, and reliable video-based assessments of technical surgical skills in the operating room
Ann Surg
Randomized clinical trial of virtual reality simulation for laparoscopic skills training
Br J Surg
The Eindhoven laparoscopic cholecystectomy training course – improving operating room performance using virtual reality training: results from the first E.A.E.S. accredited virtual reality trainings curriculum
Surg Endosc
VR to OR: a review of the evidence that virtual reality simulation improves operating room performance
World J Surg
Formula for a successful laparoscopic skills curriculum
Arch Surg
Training tomorrow's surgeons: what are we looking for and how can we achieve it?
ANZ J Surg
Cited by (13)
Surgical training with phantoms in strabismus surgery
2023, Archivos de la Sociedad Espanola de OftalmologiaA scoping review of assessment methods of competence of general surgical trainees
2023, SurgeonCitation Excerpt :In addition, the use of entrustable professional activities (EPAs) has also been cited in supporting improved autonomy.4,21 Overall, the literature suggests that graduated autonomy needs to be established to allow for progressive clinical and operative competence and this ideally would be done in an objective measured way, to ensure the safe graduation of independent surgeons at the end of training.4,8,21,29,30 Chen et al.29 proposed a set of “Performance-Impression-Characteristics-Knowledge- Situation” questions to be used as standard prior to any observation of skills within a theatre setting.
The Lake Wobegon effect is real: All general surgery residents appear to be better than average
2022, Surgery in Practice and ScienceValue of Standardized Testing in Surgical Training
2021, Surgical Clinics of North AmericaPerformance assessment - The knowledge, skills and attitudes of surgical performance
2020, Seminars in Pediatric SurgeryCitation Excerpt :As many training programs are transitioning to competency-based education, the use of entrustable professional activities (EPAs) has also been advocated as a method to evaluate technical and non-technical skills. This stems from the fact that during EPAs, evaluators are frequently called into making a judgment statement regarding trainees with respect to “entrustability” (for example “do I trust this trainee to independently and safely assess and discharge this ER patient?”).12 Lastly, NOTSS (Non-Technical Skills for Surgeons) has been developed as a tool specific for the assessment of non-technical skills of surgeons, including interpersonal skills (communication, teamwork and leadership) as well as cognitive skills (situation awareness and decision-making).
Resident evaluations: what are the predictors of future negative outcomes in surgical education?
2023, European Surgery - Acta Chirurgica Austriaca