Does quantity ensure quality? Standardized OSCE-stations for outcome-oriented evaluation of practical skills at different medical faculties

https://doi.org/10.1016/j.aanat.2017.03.006Get rights and content

Abstract

Background

Practical skills are often assessed using Objective Structured Clinical Skill Exams (OSCE). Nevertheless, in Germany, interchange and agreement between different medical faculties or a general agreement on the minimum standard for passing is lacking.

Methods

We developed standardized OSCE-stations for assessing structured clinical examination of knee and shoulder joint with identical checklists and evaluation standards. These were implemented into the OSCE-course at five different medical faculties. Learning objectives for passing the stations were agreed beforehand. At each faculty, one reference examiner scored independently of the local examiner.

Outcome of the students at the standardized station was compared between faculties and correlated to their total outcome at the OSCE, to their results at the Part One of the National Medical Licensing Examination as a reference test during medical studies and to their previous amount of lessons in examining joints.

Results

Comparing the results of the reference examiner, outcome at the station differed significantly between some of the participating medical faculties. Depending on the faculty, mean total results at the knee-examination-station differed from 64.4% to 77.9% and at the shoulder-examination-station from 62.6% to 79.2%. Differences were seen in knowledge-based items and also in competencies like communication and professional manner. There was a weak correlation between outcome at the joint-examination-OSCE-station and Part One of the National Medical Licensing Examination, and a modest correlation between outcome at the joint-examination-station and total OSCE-result. Correlation to the previous amount of lessons in examining joint was also weak.

Conclusion

Although addressing approved learning objectives, different outcomes were achieved when testing a clinical skill at different medical faculties with a standardized OSCE-station. Results can be used as a tool for evaluating lessons, training and curricula at the different sites. Nevertheless, this study shows the importance of information exchange and agreement upon certain benchmarks and evaluation standards when assessing practical skills.

Introduction

High quality medical competencies are mainly achieved by good medical education. Therefore every effort to increase the level of competence of future doctors should be based on improving medical education. In Germany, the contents of learning are oriented toward the competence based national catalogue of learning objectives (NKLM) (GMA, 2015), which has recently been established. In alignment with the NKLM, the surgical working group for education (CAL), a section of the German Society for Surgery (DGCH) has developed the concerted catalogue of learning objectives for the surgical fields (Kadmon et al., 2013). It contains clinical aspects, competencies and skills with defined levels and milestones of competence.

In spite of having the opportunity to refer to this catalogue of learning objectives it seems that teaching practical skills is rather reliant on the particular teacher and differs between faculties as well as internationally even in the case of identical tests, checklists and central examiners (Chesser et al., 2009, Stern et al., 2005). In order to set a comparable standard level of competence in practical skills, not only learning objectives, but also standardized assessments are needed. Theoretical knowledge is easily assessed by standardized exams like the national medical licensing examination in Germany. Evaluating practical skills is rather challenging. In Anglo-American countries the Objective Structured Clinical Skills Exam (OSCE) (Hubbard et al., 1965) is a well-established form of assessing practical competencies and increasingly also used at German medical schools. Although OSCE achieves a high reliability even at different sites and also in different languages (Brailovsky et al., 1992), other studies show the variety of contents, checklists and outcome as well as a high dependence on the examiner at different medical schools (Chesser et al., 2009, Makinen et al., 2010). On the other hand, some studies show that practical skills are not necessarily learned in a sufficient way during the later residency programs (Griesser et al., 2012), thus it makes sense to ensure a certain standard of practical competency either during or after medical school. The United States of America, Canada and Switzerland have already introduced assessments of practical competencies in their national medical licensing examination. Since 2004 the United States Medical Licensing Examination (USMLE) includes evaluation of practical skills in the “Step 2” of the USMLE and in the State Licensing Examination (FLE) of Switzerland these clinical skills have been assessed with standardized OSCE-stations since 2011 (De Champlain et al., 2006, Guttormsen et al., 2013). American studies have shown a high correlation of outcome in “Step 2” of USMLE and performance during the later internship (Taylor et al., 2005). Experiences from these countries also indicate the great amount of preparatory work and exchange between the different faculties, which was needed in order to set a standard which is broadly accepted by all medical faculties (Guttormsen et al., 2013). The relevance of assessed content as well as the particular benchmark has to be ensured and it also has to be assured that contents are taught at each faculty. In Germany, we are far from any agreement or even exchange on standards in evaluation of clinical and practical skills. Each medical faculty sets its own standard for the level of practical competencies which is needed at the end of medical school for working safely with patients. This lack of exchange and agreements upon certain standards is partly caused by still rudimentary networks for improvements in medical education when compared to experimental medical research networks. Furthermore, each faculty has got its own sovereignty when it comes to implementation of an own curriculum. These difficulties can only be confronted and overcome by presenting persuasive results of studies and increasing the power of educational networks. A good international example of a well-functioning exchange between medical faculties is the Australian Collaboration for Clinical Assessment in Medicine (ACCLAiM), which is supported by the Australian government. OSCE-stations are presented on online-databases for free use at all Australian medical schools and trainings for setting-up and evaluating OSCEs are offered. Thus this collaboration helps to set an Australian benchmark in assessing clinical and practical skills. The Australian medical faculties are getting regular feedback when using the platform and therefore are able to analyse their own curriculum for strengths and weaknesses.

Looking at the Australian example, it seems logical for Germany to enforce a similar network, which promotes exchange and the know-how for setting up OSCE-stations for assessing practical competencies. This would then lead to an outcome-oriented evaluation of performance and curricula with comparable criteria for assessment. Some fundamental work is already being done in establishing a network between different medical faculties and by founding the Umbrella Consortium for Assessment Networks (UCAN). Cooperation and sharing of resources in the context of exams and assessments are main pillars of the consortium. For working efficiently in a network and setting standards it is nevertheless necessary to perform precise studies with the help of these networks to ensure the outcome-orientated evaluation of competencies and curriculum and to enlarge networks and share resources in a rational manner.

The aim of this study has been to create standardized OSCE-stations with identical checklists, which are then established at different medical faculties. Effects of different examiners should be ruled out, thus we did not appoint several central examiners but only one reference examiner to score at each faculty. Only the results scored by this reference examiner were used for comparison of outcome at the different medical faculties. Results of all students were compared and with respect to gender. Furthermore results at the standardized station were correlated to the total surgical OSCE-result, as well as to a reference exam and to the number of prior lessons in the particular topic of joint examination.

Section snippets

Establishment of standardized OSCE-stations

Previous observation of surgical OSCEs at different medical faculties established stations for testing structured examination of knee and shoulder joint as two practical skills, which were taught and assessed at every faculty. Also these skills do not need elaborate technical support or a skills lab to be taught. Thus we created a basic form of structured examination of knee joint and one of shoulder joint. The time allotted for demonstration of the examination was five minutes. Checklists

Results

In two of the five participating faculties the surgical OSCE was performed simultaneously in four parallel series. Because we had only one reference examiner, only a proportion of the whole semester cohort could be tested at these two sites for logistical reasons. At the other three sites, all students performed either the knee- or shoulder-joint-examination and could therefore be assessed by the reference examiner. Finally, 180 students at site 1, 147 students at site 2, 137 at site 3 and 31

Discussion

Agreement on a definitive standard for practical skills and competencies other than knowledge at the end of medical studies is difficult. Nevertheless, the roles of a doctor emcompasses much more than being a medical expert (communicator, scholar, manager, professional, collaborator, health advocate), as the Royal College of Physicians and Surgeons of Canada stated in their framework (RoyalCollege, 2005). In order to ensure that medical students have learned to fulfil these various roles, they

Conclusion

Our study shows differences in skills between medical faculties that cannot be explained fully by the setting or the examiner or the checklists. Also it definitely shows that quantity of teaching does not necessarily lead to higher outcome at the cumulative testing of practical skills. Thus it provides each faculty with feedback on needed modifications to their courses or curricula. It further underlines the idea of a platform for sharing of OSCE-stations in order to agree on quality standards,

Acknowledgement

This study was funded by the German Federal Ministry of Education and Research (grant 01PL12038A-C).

References (34)

  • M.J. Griesser et al.

    Implementation of an objective structured clinical exam (OSCE) into orthopedic surgery residency training

    J. Surg. Educ.

    (2012)
  • M.L. Mitchell et al.

    Application of best practice guidelines for OSCEs—an Australian evaluation of their feasibility and value

    Nurse Educ. Today

    (2015)
  • BMG, Ministry of Health, Federal Law Gazette Vol. 2012, No 34. [Bundesministerium für Gesundheit, Bundesgesetzblatt...
  • K.A. Boursicot et al.

    Standard setting for clinical competence at graduation from medical school: a comparison of passing scores across five medical schools

    Adv. Health Sci. Educ. Theory Pract.

    (2006)
  • K.A. Boursicot et al.

    Using borderline methods to compare passing standards for OSCEs at graduation across three medical schools

    Med. Educ.

    (2007)
  • C.A. Brailovsky et al.

    A large-scale multicenter objective structured clinical examination for licensure

    Acad. Med.

    (1992)
  • A. Chesser et al.

    Sources of variation in performance on a shared OSCE station across four UK medical schools

    Med. Educ.

    (2009)
  • L.B. Craig et al.

    Obstetrics and gynecology clerkship for males and females: similar curriculum, different outcomes?

    Med. Educ. Online

    (2013)
  • M.H. Davis

    OSCE: the Dundee experience

    Med. Teach.

    (2003)
  • A. De Champlain et al.

    Assessing the underlying structure of the United States Medical Licensing Examination Step 2 test of clinical skills using confirmatory factor analysis

    Acad. Med.

    (2006)
  • M.L. Denney et al.

    MRCGP CSA: are the examiners biased, favouring their own by sex, ethnicity, and degree source?

    Br. J. Gen. Pract.

    (2013)
  • L. Eberhard et al.

    Analysis of quality and feasibility of an objective structured clinical examination (OSCE) in preclinical dental education

    Eur. J. Dent. Educ.

    (2011)
  • R.M. Epstein

    Assessment in medical education

    N. Engl. J. Med.

    (2007)
  • GMA, German Association for Medical Association

    Competence based national catalogue of learning objectives

    (2015)
  • P. Grand'Maison et al.

    Content validity of the Quebec licensing examination (OSCE). Assessed by practising physicians

    Can. Fam. Physician

    (1996)
  • S. Guttormsen et al.

    The new licencing examination for human medicine: from concept to implementation

    Swiss Med. Wkly.

    (2013)
  • B. Hodges

    Validity and the OSCE

    Med. Teach.

    (2003)
  • Cited by (12)

    View all citing articles on Scopus

    This paper belongs to the special issue Medical Education 2018.

    1

    Present address: Institute for Medical and Pharmaceutical Tests, Große Langgasse 8, 55116 Mainz, Germany.

    2

    Present address: Institute for Medical Education and Educational Research, Josef-Schneider-Str. 2/D6, 97080 Würzburg, Germany.

    View full text