Intelligence tests are used in a variety of settings such as schools, clinics, and courts to assess the intellectual capacity of individuals of all ages. Intelligence tests are used to make high-stakes decisions such as special education placement, employment, eligibility for social security services, and determination of the death penalty.…

We explored three approaches to resolving or re-scoring constructed-response items in mixed-format assessments: rater agreement, person fit, and targeted double scoring (TDS). We used a simulation study to consider how the three approaches impact the psychometric properties of student achievement estimates, with an emphasis on person fit. We found…

Variation in examiner stringency is an ongoing problem in many performance settings such as in OSCEs, and usually is conceptualised and measured based on scores/grades examiners award. Under borderline regression, the standard within a station is set using checklist/domain scores and global grades acting in combination. This complexity requires a…

This paper first analyzed two studies on rater factors and rating criteria to raise the problem of rater agreement. After that the author reveals the causes of discrepencies in rating administration by discussing rater variability and rater bias. The author argues that rater bias can not be eliminated completely, we can only reduce the error to a…

Physician-patient communication is a clinical skill that can be learned and has a positive impact on patient satisfaction and health outcomes. A concerted effort at all medical schools is now directed at teaching and evaluating this core skill. Student communication skills are often assessed by an Objective Structure Clinical Examination (OSCE).…

Analyzed 56 Wechsler Intelligence Scale for Children-Revised protocols completed by 1 certified and 8 licensed practitioners to examine administration and scoring mistakes. Observed numerous mistakes (failure to record examinee responses, assigning too few or too many points to answers, inappropriate questioning, and failure to obtain correct…

Examined 60 Wechsler Intelligence Scale for Children-Third Education (WISC-III) protocols administered by graduate students in training to obtain preliminary data on the frequency and types of administration and scoring errors that examiners commit. The five most frequent errors included failure to query, failure to record response verbatim,…

Analyzed for examiner errors 55 Wide Range Achievement Test-Revised (WRAT-R) protocols completed by 9 practitioners for metropolitan school district. All practitioners made errors, which occurred on 95 percent of protocols and averaged 3.0 errors per protocol. Most frequent errors included failures to obtain correct ceiling or basal, and failures…

Five of the common misuses of tests are reviewed: (1) acceptance of the test title as an accurate and complete description of the variable being measured (failure to examine the manual and the items carefully to know the specific aspects to be tested can result in misuse through selection of an inappropriate test for a particular purpose or…

Clinical psychology graduate students (N=14) administered Wechsler Intelligence Scale for Children-Revised. Found numerous scoring and mechanical errors that influenced full-scale intelligence quotient scores on two-thirds of protocols. Particularly prone to error were Verbal subtests of Vocabulary, Comprehension, and Similarities. Noted specific…

Studied the effects of experience in administration and scoring of the Wechsler Intelligence Scale for Children (Revised) on types of examiner errors. Results showed total score and administrative error rates dropped significantly with experience and feedback, but response scoring errors, mathematical errors, and IQ errors were not reduced…

Whether adequate levels of interrater reliability could be obtained on a national, standardized examination using one examiner per observation was studied with 101 paired candidate observations on an examination for optometry. Results indicate that psychometrically sound judgments can be obtained with one examiner. (SLD)