Cultural consensus theory (CCT) is a data aggregation technique with many applications in the social and behavioral sciences. We describe the intuition and theory behind a set of CCT models for continuous type data using maximum likelihood inference methodology. We describe how bias parameters can be incorporated into these models. We introduce…

Contrary to previous research on sequential ratings of student performance, this study found that professional essay raters of a large-scale standardized testing program produced ratings that were drawn toward previous ratings, creating an assimilation effect. Longer intervals between the two adjacent ratings and higher degree of agreement with…

The overclaiming technique is a novel assessment procedure that uses signal detection analysis to generate indices of knowledge accuracy (OC-accuracy) and self-enhancement (OC-bias). The technique has previously shown robustness over varied knowledge domains as well as low reactivity across administration contexts. Here we compared the OC-accuracy…

GURU is a computer program developed to analyze data generated by open-ended question techniques such as ECHO or other semistructured data collection techniques in which data are categorized. The program provides extensive descriptive statistics and allows extensive flexibility in comparing data. (Author/JKS)

This study examined the reliability of holistic scoring for a sample of essays written during the Spring 1985 MCAT administration. Analysis of variance techniques was used to estimate the reliability of scoring and to partition score variance into that due to level differences between papers and to context-specific factors. (Author/LMO)

Three essay questions were administered to junior education major college students. Factor analysis of the ratings showed that content played a large role in the students' responses, yielding dominant first order factors. Generalizability theory, used to examine the reliability of students' ratings, showed the need for more raters and questions.…

An investigation of certain chance and systematic factors affecting the grades assigned by 17 law school professors to the answers of 79 law students to a typical essay question. (Authors/MB)

A validational procedure was used, which involved a matrix of intercorrelations among tests reresenting four areas of Chartered Life Underwriter content knowledge, each measured by objective multiple-choice and essay methods. Results indicated that the two methods of measuring the same trait yielded fairly consistent estimates of content…

Ratings of student performance for two essay questions rendered by professors of English and by professors in other disciplines were compared for reliability and concurrent validity. It was concluded that the reliability and validity of the ratings of the two groups were nearly comparable. (Author/BW)

A number of studies have found that essays are scored higher when preceded by poor quality scripts than when preceded by good quality scripts. This study investigated the effects of scoring procedures designed to reduce the influence of context. Context effects were found irrespective of the scoring procedure used. (Author/PN)

Test-retest correlations can lead to biased reliability estimates when there is instability of true scores and/or when measurement errors are correlated. Using three administrations of the Test of Standard Written English and essay ratings, an analysis is demonstrated which separates true score instability and correlated errors. (Author/BW)

This paper elaborates on the work of Cardinet, and others, by clarifying some points regarding calculations, specifically with reference to existing computer programs, and by presenting illustrative examples of the calculation and interpretation of several generalizability coefficients from a complex six-facet (factor) design. (Author/RL)

The essay-type items of postgraduate achievement examinations in psychology for students at the University of Peshawar were classified, and the relative frequencies of the items were compared with those for psychology examinations administered at a British university. The dimensions of Analysis and Synthesis appeared to be underrepresented at…

Research reported in this paper considered the construct validity of a trial essay administered in 1985-87 Medical College Admission Test (MCAT). The addition of the essay caused the non-science factor observed in previous MCAT research to be more strongly defined. (Author/LMO)

For a sample of more than 900 college students and subsamples of at least 50 students in each of nine departments, concurrent validity coefficients were determined between scores on several examinations reflecting competencies in standard written English and an external criterion measure of cumulative college grade point average. (Author/CTM)