Standard setting is a fairly widespread activity in educational and psychological measurement, but there is no formal psychometric theory to guide the development of standard setting methodology. This paper presents a conceptual framework for such a psychometric theory and uses the conceptual framework to analyze a number of methods for setting…

The objectives of this study were to determine whether student writing portfolios could be rated reliably by trained judges; study the effects on student ratings of the differential leniency of the judges; and ascertain the effects of writing-prompt difficulty and its interactions with rater leniency. Writing samples from 127 students in grades 3,…

In circumstances where judges are required to make ratings of performance, it is usually required to have two or more raters who are trained to agree on independent ratings of the same performance. It is suggested that such a requirement may produce the paradox of attenuation associated with item analysis, in which too high a correlation between…

College debaters who go to law school are often surprised by the differences between the processes that take place in the court of reason and the process that takes place in the court of law. The court of reason relies mainly on authoritative testimony, while the court of law relies on direct evidence. Evidence in the court of reason is either…