Proposes a highly structured six-step employment interviewing technique which includes asking the same questions, consistently administering the process to all candidates, and having an interview panel. Results of a field study of 243 job applicants using this technique demonstrated interrater reliability, predictive validity, test fairness for…

A performance-based student teacher evaluation process was investigated to see if halo and leniency errors could be eliminated. Results are presented. (MT)

A study revealing wide variation in the grading of electronics engineering test items by different evaluators has implications for evaluator and test item selection, analysis and manipulation of grades, and the use of numerical methods of assessment. (MSE)

Type and comprehensiveness of training were experimentally manipulated (N=128) to study their effects on the reliability and validity of rated counselor empathy. Implications for observer training are discussed. (Author)

Nearly all of the research on standard setting focuses on different standard setting methods rather than the interaction of group members and the instructions given to group members. This study explored the effect of deliberation style and the requirement to reach consensus on the passing score, on rater satisfaction, and on postdecision…

This paper illustrates a method to study rater severity across exam administrations. A multi-facet Rasch model defined the ratings as being dominated by four facets: examinee ability, rater severity, project difficulty, and task difficulty. Ten years of data from administrations of a histotechnology performance assessment were pooled and analyzed…

This empirical study used data from the Reading: Basic Understanding section of the New Standards English Language Arts Examination. Data were collected for 3,200 high school students randomly selected from those who took the examination. The resulting sample had 16 raters who scored 200 students each, with each student rated by only 1 rater. The…

This paper provides both logical and empirical evidence to justify the use of an item mapping method for establishing passing scores for multiple-choice licensure and certification examinations. After describing the item-mapping standard setting process, the paper discusses the theoretical basis and rationale for this newly developed method and…

An interjudge reliability test was conducted to evaluate questionnaires used in the surveillance of residential care institutions in Israel. Results from 32 institutions (evaluated by two surveyor teams--one social worker and 1 nurse per team) and the variance in reliability were used to improve the questionnaires and their administration. (SLD)

Reviews the importance of intercoder agreement for content analysis in mass communication research. Describes several indices for calculating this type of reliability (varying in appropriateness, complexity, and apparent prevalence of use). Presents a content analysis of content analyses reported in communication journals to establish how…

Employed triangulation strategy for assessing family interaction, involving family members, therapist, and coders independently viewing videotapes. Found weak agreement between paired assessments within family triad, and within therapist-coder dyad. Findings suggest that methodological and/or scaling strategies designed to maximize agreement may…

Interviewed Israeli students to assess the Adult Attachment Interview's test-retest reliability and effects of the interviewers on the interview itself. Information about subjects' memory and intellectual abilities was obtained from external sources. Found a high degree of interrater and test-retest reliabilities, irrespective of interviewers.…

Case means and case failures in performance-based medical student evaluations were examined to evaluate the consistency of ratings made by two or more standardized patients (SPs) simulating the same case. Results demonstrate a need for caution in interpreting scores obtained from a case checklist completed by multiple SPs. (Author/MSE)

The Neurobehavioral Assessment of the Preterm Infant instrument was developed by means of pilot, exploratory, and validation studies. The validation study tested the generalizability of results for different cohorts, test versions, hospitals, and examiners. Seven stable functions were identified: motor development; scarf sign; popliteal angle;…

Analysis of a study of sex bias in undergraduate student project evaluations revealed evidence of bias that was overlooked by the researchers. Research methodology and interpretation are discussed further. (MSE)