Abstract
In 1994 and 1995, the Medical Council of Canada used an innovative approach to set the pass mark on its large scale, multi-center national OSCE which is designed to assess basic clinical and communication skills in physicians in Canada after 15 months of post-graduate medical training. The goal of this article is to describe the new approach and to present the experience with the method during its first two years of operation. The approach utilizes the global judgments of the physician examiners at each station to identify the candidates with borderline performances. The scores of the candidates whose performances are judged to be borderline are summed for each station, yielding an initial passing score for all stations and then the examination as a whole. The latter score is then adjusted upward one standard error of measurement for the final passing score and is used as one of the criteria to pass the examination. Based on the results to date, the new approach has worked well. The advantages, disadvantages and areas of possible refinement for the approach are reviewed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Cohen DF, Colliver JE, Marcy MS, Fried ED and Swartz MH. (1996). Psychometric Properties of a Standardized-Patient Checklist and Rating-Scale Form Used to Assess Interpersonal and Communication Skills. Academic Medicine 71(Supplement): S87-S89.
Cohen R, Rothman AI, Bilan S and Ross J. (1996). Analysis of the Psychometric Properties of Eight Administrations of an Objective Structured Clinical Examination Used to assess international Medical Graduates. Academic Medicine 71(Supplement): S22-S24.
Cohen R et al. (1988). A Comprehensive Assessment of Graduates of Foreign Medical Schools. Annals of the Royal College of Physicians and Surgeons of Canada 21: 505-509.
Colliver JA, Verhulst SJ, Williams RG and Norcini JJ. (1989). Reliability of performance on standardized patient cases. A comparison of consistency measures based on generalizability theory. Teaching and Learning in Medicine 1: 31-37.
Cuisimano MD. (1996). Standard Setting in Medical Education. Academic Medicine 71(Supplement): S112-S120.
Dauphinee WD and Blackmore DE. (1995). Setting Minimal Passing Standards for the Qualifying Examination of the Medical Council of Canada: The Transition from Norm-Referencing to Criterion Referencing. In: Rothman AI and Cohen R, eds. Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto ON. University of Toronto Book Store, pp. 245-247.
Dauphinee WD, Baumber JS, Reznick RK, et al. (1995). Rationalizing a Licensing Examination System: The New Examinations of the Medical Council of Canada. In: Rothman AI and Cohen R, (eds). Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto, ON. University of Toronto, pp. 26-28.
Gleeson F. (1995). World History and Geography of the OSCE Usage in the British Isles and Ireland. In: Rothman AI (ed). Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto, Ontario, Canada. University of Toronto Bookstore Custom Publishing, pp. 292-298.
Grand'Maison P. (1994). OSCEs in Canada: Leadership for More than 15 Years. In: Rothman AI (ed) Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto, Ontario, Canada. University of Toronto Bookstore Custom Publishing, pp. 303-305.
Grand'Maison P, Lescop J, Rainsberry P and Brailovsky CA. (1992). Large Scale Use of Objective Structured Clinical Examination for Licensing Family Physicians. Canadian Medical Association Journal 146: 1735-1740.
Gray JD. (1996). Global Rating Scales in Residency Education. Academic Medicine 71(Supplement): S55-S63.
Harden RM, Stevenson M, Downie WW and Wilson GV. (1975). Assessment of Clinical Competence Using Objective Structured Examination. British Medical Journal 1: 447-451.
Klass, DJ. (1994). “High Stakes” Testing of Medical Students Using Standardized Patients. Teaching and Learning in Medicine 6: 28-32.
Klass D, Hazzard T, Kopelow M, Tamblyn R and Barrows H. (1987). Portability of a Multiple Station Performance Based Assessment of Clinical Competence. In: Hart IR and Harden RM (eds), Further Developments in Assessing Clinical Competence. Montreal, Quebec. Can-Heal Publications, pp. 434-439.
Lescop J, Grand'Maison P, and Brailovsky C. (1995). Setting minimal standards in the Quebec Licensing Examination. In: Rothman AI (ed). Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto, Ontario, Canada. University of Toronto Bookstore Custom Publishing, pp. 351-354.
Martin J, Reghr G, Reznick R, MacRae H, Murnaghan J, Hutchison C and Brown M. (1997). An objective structured assessment of technical skill (OSATS) for surgical residents. British Journal of Surgery 84: 273-278.
Martin JA, Reznick RK, Rothman A et al. (1996). Who Should Rate Candidates in an Objective Structured Clinical Examination? Academic Medicine 71: 170-175.
Newble D, Hoare A and Elmslie R. (1981). The Validity and Reliability of a New Examination of Clinical Competence of Medical Students. Medical Education 17: 165-171.
Norcini JJ, Stillman PL, Sutnick AI, Regan MB, Haley HL et al. (1993). Scoring and Standard Setting With Standardized Patients. Evaluation & The Health Professions 16: 322-332.
Norman G. (1996). Concluding Comments. Seventh Ottawa Conference on Medical Education. Maastricht (in press).
Norman GR. (1994). Theoretical and psychometric considerations. In: Task Force of the Evaluation Committee, Royal College of Physicians and Surgeons of Canada. Report on the evaluation system for specialist certification.
Norman GR, Feightner JW. (1981). A comparison of behavior on simulated and patients management problems. Medical Education 15: 26-32.
Page G and Bordage G. (1995). The Medical Council of Canada's Key Features Project: A More Valid Written Examination of Clinical Decision-Making Skills. Academic Medicine 70: 104-110.
Page G, Bordage G and Allen T. (1995). Developing Key-Feature Problems and Examinations to Assess Clinical Decision-making Skills. Academic Medicine 70: 194-201.
Petrusa ER. (1995). OSCEs and Other Multiple-Station SP-Based Examinations in the United States. In: Rothman AI (ed) Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto, Ontario, Canada. University of Toronto Book Store Custom Publishing, pp. 298-300.
Petrusa ER et al. (1991). A Multi-Institutional Trial of an Objective Structured Clinical Examination. Teaching and Learning in Medicine 3: 86-94.
Reznick R, Cohen R. (1995). An objective structured clinical examination for senior residents in Surgery: Testing the high end. In: Rothman AI and Cohen R (eds), Proceedings of the Sixth Ottawa Conference on Medical Education. Toronto ON. University of Toronto Book Store, pp. 203-207.
Reznick R, Smee S, Rothman A, et al. (1992). An objective structured clinical examination for the licentiate: report of the pilot project of the Medical Council of Canada. Academic Medicine 67: 487-494.
Reznick RK, Blackmore DE, Dauphinee WD, et al. (1996). Large-Scale High Stakes Testing with an OSCE: Report from the Medical Council of Canada. Academic Medicine 71(Supplement): S19-S21.
Reznick RK, Blackmore D, Cohen R, et al. (1993). An objective clinical examination for the Licentiate of the Medical Council of Canada; from research to reality. Academic Medicine 68(Suppl. 10): S4-S6.
Ross LP, Clauser BE, Margolis MJ, Orr NA and Klass DJ. (1995). An Expert-Judgment Approach to Setting Standards for a Standardized-Patient Examination. Academic Medicine 71: S4-S6.
Stillman P, Swanson D, Regan MB, Philbin MM et al. (1991). Assessment of clinical skills of residents using standardized patients. A follow-up study and recommendations. Annals of Internal Medicine 114: 393-401.
Stillman PL et al. (1986). Assessing Clinical Skills of Residents with Standardized Patients. Annals of Internal Medicine 105: 762-771.
Sutnick AI, Freedman M, Stillman PL, Norcini JJ and Wilson MP. (1994). International Use of Standardized Patients. Teaching and Learning in Medicine 6: 33-35.
Swanson DB and Norcini JJ. (1989). Factors Influencing Reproducibility of Tests Using Standardized Patients. Teaching and Learning in Medicine 1: 158-166.
Tamblyn RM, Klass DK, Schnabl GK and Kopelow ML. (1990). Factors Associated with the Accuracy of Standardized Patient Presentations. Academic Medicine 65(Supplement): S55-S56.
Travis TA, Colliver JA, Robbs RS, Barnhart AJ, Barrows HS et al. (1996). Validity of a Simple Approach to Scoring and Standard Setting for Standardized-Patient Cases in an Examination of Clinical Competence. Academic Medicine 71(Supplement): S84-S86.
Vu NV, Steward DE and Marcy M. (1987). An Assessment of Consistency and Accuracy of Standardized Patients' Simulations. Journal of Medical Education 62: 1000-1002.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Dauphinee, W.D., Blackmore, D.E., Smee, S. et al. Using the Judgments of Physician Examiners in Setting the Standards for a National Multi-center High Stakes OSCE. Adv Health Sci Educ Theory Pract 2, 201–211 (1997). https://doi.org/10.1023/A:1009768127620
Issue Date:
DOI: https://doi.org/10.1023/A:1009768127620