Requiring a Consistent Unit of Scale Between the Responses of Students and Judges in Standard Setting

One of the best-known methods for setting a benchmark standard on a test is that of Angoff and its modifications. When scored dichotomously, judges estimate the probability that a benchmark student has of answering each item correctly. As in most methods of standard setting, it is assumed implicitly that the unit of the latent scale of the responses of the judges is the same as that of the responses of the students. In an empirical example using a modified Angoff approach, this article shows that this assumption is not correct. In particular, using the dichotomous Rasch model, the article shows that the unit of the latent scale for the students is smaller than that of the judges. Accordingly, to obtain an unbiased estimate of a standard, the article shows that it is necessary to place the judges' responses onto a latent scale with the same unit as that of the students. Implications for standard setting procedures are considered, including understanding some observed difficulties reported in the literature in applying Angoff's standard setting procedure. © 2014 Copyright Taylor and Francis Group, LLC.