Angovian Methods for Standard Setting in Medical Education: Can They Ever Be Criterion Referenced?

Abstract

This paper presents a discussion of Angovian methods of standard setting – methods which are widely used with the intent of defining criterion-referenced absolute standards for tests in medical education.  Most practitioners, although purporting to pursue absolute, criterion-referenced standards, have unwittingly slipped into focussing on norm-referenced concepts of ‘borderline’ students and their predicted ability to answer assessment items in a test.  This slippage has been facilitated by a shift in language from the original concept of ‘minimally acceptable’ persons to the modern concept of ‘borderline’ persons.

The inability of university academics to predict accurately the performance of ‘borderline’ graduate-entry medical students is illustrated by presentation of data obtained from three successive cohorts of a small regional medical school during the years 2010-2012.  Other data are presented to show how student performance, both ‘borderline’ and general, can be significantly altered by switching from didactic lectures to tutorials preceded by task-based active learning.

A protocol, based on a stricter interpretation of what is meant by a ‘minimally acceptable’ person, is suggested for moving towards a more criterion-referenced standard for a test based on the curriculum’s learning objectives.  Nonetheless, the fallibility of criterion-referenced standard-setting processes means that norm-referenced relative standards may need to be brought into play to deal with anomalous grade results should they arise.

The ideal of defining an absolute criterion-referenced standard for a test, using the most commonly implemented Angovian method, is probably as least as unattainable for graduate-entry medicine as it has been previously shown to be for secondary school science.