Skip to main content

Advertisement

SpringerLink
Log in

Overview: what’s worked and what hasn’t as a guide towards predictive admissions tool development

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Admissions committees and researchers around the globe have used diligence and imagination to develop and implement various screening measures with the ultimate goal of predicting future clinical and professional performance. What works for predicting future job performance in the human resources world and in most of the academic world may not, however, work for the highly competitive world of medical school applicants. For the job of differentiating within the highly range-restricted pool of medical school aspirants, only the most reliable assessment tools need apply. The tools that have generally shown predictive validity in future performance include academic scores like grade point average, aptitude tests like the Medical College Admissions Test, and non-cognitive testing like the multiple mini-interview. The list of assessment tools that have not robustly met that mark is longer, including personal interview, personal statement, letters of reference, personality testing, emotional intelligence and (so far) situational judgment tests. When seen purely from the standpoint of predictive validity, the trends over time towards success or failure of these measures provide insight into future tool development.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Albanese, M. A., Snow, M. H., Skochelak, S. E., Huggett, K. N., & Farrell, P. M. (2003). Assessing personal qualities in medical school admissions. Academic Medicine, 78(3), 313–321. doi:10.1097/00001888-200303000-00016.

    Article  Google Scholar 

  • Bandalos, D., & Sedlacek, W. (1989). Predicting success of pharmacy students using traditional and nontraditional measures by race. American Journal of Pharmaceutical Education, 53, 145–148.

    Google Scholar 

  • Barr, D. A., Gonzalez, M. E., & Wanat, S. F. (2008). The leaky pipeline: factors associated with early decline in interest in premedical studies among underrepresented minority undergraduate students. Academic Medicine, 83(5), 503–511.

    Article  Google Scholar 

  • Barrick, M. R., & Mount, M. K. (1991). The Big 5 personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1–26. doi:10.1111/j.1744-6570.1991.tb00688.x.

    Article  Google Scholar 

  • Behling, C. (1998). Employee selection: Will intelligence and conscientiousness do the job? Academy of Management Executive, 12(1), 77–86.

    Google Scholar 

  • Berstein, A. D., Jazrawi, L. M., Elbeshbeshy, B., Della Valle, C. J., & Zukerman, J. D. (2002). An analysis of orthopaedic residency selection criteria. Bulletin (Hospital for Joint Diseases (New York, NY)), 61(1–2), 49–57.

    Google Scholar 

  • Boyer, S. P., & Sedlacek, W. E. (1988). Noncognitive predictors of academic success for international students: A longitudinal study. Journal of College Student Development, 29, 218–222.

    Google Scholar 

  • Brothers, T. E., & Wetherholt, S. (2007). Importance of the faculty interview during the resident application process. Journal of Surgical Education, 64(6), 378–385. doi:10.1016/j.jsurg.2007.05.003.

    Article  Google Scholar 

  • Brown, B., Carpio, B., & Roberts, J. (1991). The use of an autobiographical letter in the nursing admissions process: Initial reliability and validity. Canadian Journal of Nursing Research, 23(2), 9–20.

    Google Scholar 

  • DeLisa, J. A., Jain, S. S., & Campagnolo, D. I. (1994). Factors used by physical medicine and rehabilitation residency training directors to select their residents. American Journal of Physical Medicine and Rehabilitation, 73(3), 152–156. doi:10.1097/00002060-199406000-00002.

    Article  Google Scholar 

  • Dickman, R. L., Sarnacki, R. E., Schimpfhauser, F. T., & Katz, L. A. (1980). Medical students from natural science and nonscience undergraduate backgrounds. Similar academic performance and residency selection. Journal of the American Medical Association, 243(24), 2506–2509. doi:10.1001/jama.243.24.2506.

    Article  Google Scholar 

  • Dirschl, D. R., & Adams, G. L. (2000). Reliability in evaluating letters of recommendation. Academic Medicine, 75(10), 1029. doi:10.1097/00001888-200010000-00022.

    Article  Google Scholar 

  • Donnon, T., Paolucci, E. O., & Violato, C. (2007). The predictive validity of the MCAT for medical school performance and medical board licensing examinations: A meta-analysis of the published research. Academic Medicine, 82(1), 100–106. doi:10.1097/01.ACM.0000249878.25186.b7.

    Article  Google Scholar 

  • Dore, K. L., Hanson, M., Reiter, H. I., Blanchard, M., Deeth, K., & Eva, K. W. (2006). Medical school admissions: Enhancing the reliability and validity of an autobiographical screening tool. Academic Medicine, 81(10, Suppl), S70–S73. doi:10.1097/01.ACM.0000236537.42336.f0.

    Article  Google Scholar 

  • Eva, K. W., & Regehr, G. (2007). Knowing when to look it up: A new conception of self-assessment ability. Academic Medicine, 82(10, Suppl), S81–S84. doi:10.1097/ACM.0b013e31813e6755.

    Article  Google Scholar 

  • Eva, K. W., Reiter, H. I., Rosenfeld, J., & Norman, G. R. (2004). The ability of the multiple mini-interview to predict preclerkship performance in medical school. Academic Medicine, 79(10, Suppl), S40–S42. doi:10.1097/00001888-200410001-00012.

    Article  Google Scholar 

  • Eva, K.W., Reiter, H. I., Trinh, K., Wasi, P., Rosenfeld, J., & Norman, G.R. (2009). Predictive validity of the multiple mini-interview for undergraduate and postgraduate admissions decisions. Medical Education (accepted for publication).

  • Ferguson, E., Sanders, A., O’Hehir, F., & James, D. (2000). Predictive validity of personal statement and the role of the five factor model of personality in relation to medical training. Journal of Occupational and Organizational Psychology, 73, 321–344. doi:10.1348/096317900167056.

    Article  Google Scholar 

  • Ferguson, E., James, D., O’Hehir, F., & Sanders, A. (2003). Pilot study of the roles of personality, references, and personal statements in relation to performance over the five years of a medical degree. British Medical Journal, 326, 429–432. doi:10.1136/bmj.326.7386.429.

    Article  Google Scholar 

  • Flexner, A. (1910). Medical education in the United States and Canada. Bulletin number four. New York, NY: Carnegie Foundation for the Advancement of Teaching.

    Google Scholar 

  • Fuertes, J. N., & Sedlacek, W. E. (1994). Predicting the academic success of Hispanic university students using SAT scores. College Student Journal, 28, 350–352.

    Google Scholar 

  • Fuertes, J. N., & Sedlacek, W. E. (1995). Using noncognitive variables to predict the grades and retention of Hispanic students. College Student Affairs Journal, 14, 30–36.

    Google Scholar 

  • Gough, H. G. (1978). Some predictive implications of premedical scientific competence and preferences. Journal of Medical Education, 53(4), 291–300.

    Google Scholar 

  • Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical statistical controversy. Psychology, Public Policy, and Law, 2, 293–323.

    Article  Google Scholar 

  • Hanson, M. D., Dore, K. L., Reiter, H. I., & Eva, K. W. (2007). Medical school admissions: Revisiting the veracity and independence of completion of an autobiographical screening tool. Academic Medicine, 82, S8–S11. doi:10.1097/ACM.0b013e3181400068.

    Article  Google Scholar 

  • Huff, K. L., & Fang, D. (1999). When are students most at risk of encountering academic difficulty? A study of the 1992 matriculants to US medical schools. Academic Medicine, 74(4), 454–460. doi:10.1097/00001888-199904000-00047.

    Article  Google Scholar 

  • Huff, K. L., Koenig, J. A., Treptau, M. M., & Sireci, S. G. (1999). Validity of MCAT scores for predicting clerkship performance of medical students grouped by sex and ethnicity. Academic Medicine, 74(10, Suppl), S41–S44. doi:10.1097/00001888-199910000-00035.

    Article  Google Scholar 

  • Huffcutt, A. I., & Arthur, W., Jr. (1994). Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. The Journal of Applied Psychology, 79(2), 184–190. doi:10.1037/0021-9010.79.2.184.

    Article  Google Scholar 

  • Julian, E. R. (2005). Validity of the Medical College Admission Test for predicting medical school performance. Academic Medicine, 80(10), 910–917. doi:10.1097/00001888-200510000-00010.

    Article  Google Scholar 

  • Kirchner, G. L., & Holm, M. B. (1997). Prediction of academic and clinical performance of occupational therapy students in an entry-level master’s program. The American Journal of Occupational Therapy., 51, 775–779.

    Google Scholar 

  • Kirchner, G. L., Stone, R. G., & Holm, M. B. (2000). Use of admission criteria to predict performance of students in an entry-level master’s program on fieldwork placements and in academic courses. Occupational Therapy in Health Care, 13(1), 1. doi:10.1300/J003v13n01_01.

    Article  Google Scholar 

  • Koenig, J. A. (1992). Comparison of medical school performances and career plans of students with broad and with science-focused premedical preparation. Academic Medicine, 67(3), 191–196. doi:10.1097/00001888-199203000-00011.

    Article  Google Scholar 

  • Koenig, J. A., Sireci, S. G., & Wiley, A. (1998). Evaluating the predictive validity of MCAT scores across diverse applicant groups. Academic Medicine, 73(10), 1095–1106. doi:10.1097/00001888-199810000-00021.

    Article  Google Scholar 

  • Kreiter, C. D., & Kreiter, Y. (2007). A validity generalization perspective on the ability of undergraduate GPA and the medical college admission test to predict important outcomes. Teaching and Learning in Medicine, 19(2), 95–100.

    Google Scholar 

  • Kreiter, C. D., Yin, P., Solow, C., & Brennan, R. L. (2004). Investigating the reliability of the medical school admissions interview. Advances in Health Sciences Education: Theory and Practice, 9(2), 147–159. doi:10.1023/B:AHSE.0000027464.22411.0f.

    Article  Google Scholar 

  • Kulatunga-Moruzi, C., & Norman, G. R. (2002). Validity of admissions measures in predicting performance outcomes: The contribution of cognitive and non-cognitive dimensions. Teaching and Learning in Medicine, 14(1), 34–42. doi:10.1207/S15328015TLM1401_9.

    Article  Google Scholar 

  • Lewis, N. J., Rees, C. E., Hudson, J. N., & Bleakley, A. (2005). Emotional intelligence in medical education: Measuring the unmeasurable? Advances in Health Sciences Education, 10, 339–355. doi:10.1007/s10459-005-4861-0.

    Article  Google Scholar 

  • Lievens, F., & Sackett, P. R. (2006). Video-based versus written situational judgement tests: A comparison in terms of predictive validity. The Journal of Applied Psychology, 91(5), 1181–1188. doi:10.1037/0021-9010.91.5.1181.

    Article  Google Scholar 

  • Lievens, F., Buyse, T., & Sackett, P. R. (2005). The operational validity of a video-based situational judgement test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. The Journal of Applied Psychology, 90(3), 442–452. doi:10.1037/0021-9010.90.3.442.

    Article  Google Scholar 

  • McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. The Journal of Applied Psychology, 79, 599–616. doi:10.1037/0021-9010.79.4.599.

    Article  Google Scholar 

  • McManus, I. C., & Richards, B. (1986). Prospective study of performance of medical students during pre-clinical years. British Medical Journal, 293, 124–127.

    Article  Google Scholar 

  • Meredith, K. E., Dunlap, M. R., & Baker, H. H. (1982). Subjective and objective admissions factors as predictors of clinical clerkship performance. Journal of Medical Education, 57(10 Pt 1), 743–751.

    Google Scholar 

  • Neame, R. L., Powis, D. A., & Bristow, T. (1992). Should medical students be selected only from recent school-leavers who have studied science? Medical Education, 26(6), 433–440. doi:10.1111/j.1365-2923.1992.tb00202.x.

    Article  Google Scholar 

  • Norcini, J. J., Lipner, R. S., & Kimball, H. R. (2002). Certifying examination performance and patient outcomes following acute myocardial infarction. Medical Education, 36(9), 853–859. doi:10.1046/j.1365-2923.2002.01293.x.

    Article  Google Scholar 

  • O’Callaghan, K. W., & Bryant, C. (1990). Noncognitive variables: A key to black-American academic success at a military academy. Journal of College Student Development, 31, 121–126.

    Google Scholar 

  • O’Halloran, C. M., Altmaier, E. M., Smith, W. L., & Franken, E. A., Jr. (1993). Evaluation of resident applicants by letters of recommendation: A comparison of traditional and behavior-based formats. Investigative Radiology, 28(3), 274–277.

    Article  Google Scholar 

  • Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgement inventory as predictors of college student performance. The Journal of Applied Psychology, 89, 187–207. doi:10.1037/0021-9010.89.2.187.

    Article  Google Scholar 

  • Ozuah, P. O. (2002). Variability in Deans’ letters. Journal of the American Medical Association, 288, 1061. doi:10.1001/jama.288.9.1061.

    Article  Google Scholar 

  • Papadakis, M. A., Teherani, A., Banach, M. A., Knettler, T. R., Rattner, S. L., Stern, D. T., et al. (2005). Disciplinary action by medical boards and prior behavior in medical school. The New England Journal of Medicine, 353(25), 2673–2682. doi:10.1056/NEJMsa052596.

    Article  Google Scholar 

  • Peskun, C., Detsky, A., & Shandling, M. (2007). Effectiveness of medical school admissions criteria in predicting residency ranking four years later. Medical Education, 41(1), 57–64. doi:10.1111/j.1365-2929.2006.02647.x.

    Article  Google Scholar 

  • Powis, D. A., Neame, R. L., Bristow, T., & Murphy, L. B. (1988). The objective structured interview for medical student selection. British Medical Journal (Clinical Research Ed.), 296(6624), 765–768.

    Article  Google Scholar 

  • Ramsey, P. G., Carline, J. D., Inui, T. S., Larson, E. B., LoGerfo, J. P., & Wenrich, M. D. (1989). Predictive validity of certification by the American Board of Internal Medicine. Annals of Internal Medicine, 110(9), 719–726.

    Google Scholar 

  • Reiter, H. I., Eva, K. W., Rosenfeld, J., & Norman, G. R. (2007). Multiple mini-interviews predict clerkship and licensing examination performance. Medical Education, 41(4), 378–384.

    Article  Google Scholar 

  • Ross, C. A., & Leichner, P. (1984). Criteria for selecting residents: A reassessment. Canadian Journal of Psychiatry, 29, 681–686.

    Google Scholar 

  • Salvatori, P. (2001). Reliability and validity of admissions tools used to select students for the health professions. Advances in Health Sciences Education, 6, 159–175. doi:10.1023/A:1011489618208.

    Article  Google Scholar 

  • Schofield, W., & Garrard, J. (1975). Longitudinal study of medical students selected for admission to medical school by actuarial and committee methods. British Journal of Medical Education, 9(2), 86–90.

    Google Scholar 

  • Sedlacek, W. E. (1991). Using noncognitive variables in advising nontraditional students. National Academic Advising Association Journal, 11(1), 75–82.

    Google Scholar 

  • Sedlacek, W. E. (1996). An empirical method of determining nontraditional group status. Measurement and Evaluation in Counseling and Development, 28, 200–210.

    Google Scholar 

  • Sedlacek, W. E. (1999). Blacks on White campuses: 20 years of research. Journal of College Student Development, 40, 538–550.

    Google Scholar 

  • Sedlacek, W. E. (2003). Alternative measures in admissions and scholarship selection. Measurement and Evaluation in Counseling and Development, 35, 263–272.

    Google Scholar 

  • Sedlacek, W. E. (2004). Beyond the big test—noncognitive assessment in higher education. San Francisco: Jossey-Bass.

    Google Scholar 

  • Sedlacek, W. E., & Adams-Gaston, J. (1992). Predicting the academic success of student-athletes using SAT and noncognitive variables. Journal of Counseling and Development, 70(6), 724–727.

    Google Scholar 

  • Shapiro, R. S., Simpson, D. E., Lawrence, S. L., Talsky, A. M., Sobocinski, K. A., & Schiedermayer, D. L. (1989). A survey of sued and nonsued physicians and suing patients. Archives of Internal Medicine, 149, 2190–2196. doi:10.1001/archinte.149.10.2190.

    Article  Google Scholar 

  • Standridge, J., Boggs, K. J., & Mugan, K. A. (1997). Evaluation of the health occupations aptitude examination. Respiratory Care, 42, 868–872.

    Google Scholar 

  • Tamblyn, R., Abrahamowicz, M., Brailovsky, C., Grand’Maison, P., Lescop, J., Norcini, J., et al. (1998). Association between licensing examination scores and resource use and quality of care in primary care practice. Journal of the American Medical Association, 280(11), 989–996. doi:10.1001/jama.280.11.989.

    Article  Google Scholar 

  • Tamblyn, R., Abrahamowicz, M., Dauphinee, W. D., Hanley, J. A., Norcini, J., Girard, N., et al. (2002). Association between licensure examination scores and practice in primary care. Journal of the American Medical Association, 288(23), 3019–3026. doi:10.1001/jama.288.23.3019.

    Article  Google Scholar 

  • Tamblyn, R., Abrahamowicz, M., Dauphinee, D., Wenghofer, E., Jacques, A., Klass, D., et al. (2007). Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. Journal of the American Medical Association, 298(9), 993–1001. doi:10.1001/jama.298.9.993.

    Article  Google Scholar 

  • Taylor, P., & Small, B. (2002). Asking applicants what they would do versus what they did do: A meta-analytic comparison of situational and past behaviour employment interview questions. Journal of Occupational and Organizational Psychology, 75, 277–294. doi:10.1348/096317902320369712.

    Article  Google Scholar 

  • Ting, S. R. (1997). Estimating academic success in the first year of college for specially admitted white students: A model combining cognitive and psychosocial predictors. Journal of College Student Development, 38, 401–409.

    Google Scholar 

  • Tracey, T. J., & Sedlacek, W. E. (1985). The relationship of noncognitive variables to academic success: A longitudinal comparison by race. Journal of College Student Personnel, 26, 405–410.

    Google Scholar 

  • Tracey, T. J., & Sedlacek, W. E. (1987). Prediction of college graduation using noncognitive variable by race. Measurement and Evaluation in Counseling and Development, 19, 177–184.

    Google Scholar 

  • Tracey, T. J., & Sedlacek, W. E. (1988). A comparison of white and black student academic success using noncognitive variables: A LISREL analysis. Research in Higher Education, 27, 333–348. doi:10.1007/BF00991662.

    Article  Google Scholar 

  • Tracey, T. J., & Sedlacek, W. E. (1989). Factor structure of the noncognitive questionnaire—revised across samples of black and white college students. Educational and Psychological Measurement, 49, 637–648. doi:10.1177/001316448904900316.

    Article  Google Scholar 

  • Trail, C., Reiter, H. I., Bridge, M., Stefanowska, P., Schmuck, M., & Norman, G. (2006). Impact of field of study, college and year on calculation of cumulative grade point average. Advances in Health Sciences Education Theory and Practice (Epub ahead of print).

  • Violato, C., & Donnon, T. (2005). Does the medical college admission test predict clinical reasoning skills? A longitudinal study employing the Medical Council of Canada clinical reasoning examination. Academic Medicine, 80(10, Suppl), S14–S16. doi:10.1097/00001888-200510001-00007.

    Article  Google Scholar 

  • Wiesner, W., & Cronshaw, S. (1988). A meta-analytic investigation of the impact of interview format and degree of structure on the validity of the employment interview. Journal of Occupational Psychology, 61, 275–290.

    Google Scholar 

  • Woodward, C. A., & McAuley, R. G. (1983). Can the academic background of medical graduates be detected during internship? Canadian Medical Association Journal, 129(6), 567–569.

    Google Scholar 

  • Yens, D. P., & Stimmel, B. (1982). Science versus nonscience undergraduate studies for medical school: A study of nine classes. Journal of Medical Education, 57(6), 429–435.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. McMaster University, 1200 Main Street West, MDCL 3112, Hamilton, ON, L8N 3Z5, Canada

    Eric Siu

  2. Department of Oncology, McMaster University, 1200 Main Street West, MDCL 3112, Hamilton, ON, L8N 3Z5, Canada

    Harold I. Reiter

Authors
  1. Eric Siu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Harold I. Reiter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Harold I. Reiter.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Siu, E., Reiter, H.I. Overview: what’s worked and what hasn’t as a guide towards predictive admissions tool development. Adv in Health Sci Educ 14, 759–775 (2009). https://doi.org/10.1007/s10459-009-9160-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-009-9160-8

Keywords

Search

Navigation