Current practices of reporting a single mean intercoder agreement in meta-analysis leads to systematic bias and overestimates reliability. An alternative is recommended in which average intercoder agreement statistics are calculated within clusters of coded variables. Two studies of intercoder agreement illustrate the model. (SLD)

Two experiments studied teachers' proficiency in assessing students' higher order thinking skills. After training alone or after training plus implementation of an instructional unit on correlational thinking, teacher ratings of student samples did not correspond highly with an expert's assessment although they showed sensitivity to student age…

In this response to critiques (Mudford, Hogg and Roberts 1997, 1999) of the use of behavior states in research involving individuals with mental retardation, it is argued that the work on behavioral state analysis by Robert D. Guess has contributed to the field at the practical, empirical, and theoretical levels. (Contains references.) (CR)

Investigates how experienced and inexperienced raters score essays written by English as a Second Language (ESL) students on two different prompts. Shows that the inexperienced raters were more severe than the experienced raters on one prompt but not on the other prompt, and that differences between the two groups of raters were eliminated…

Presents a model-based approach to rater reliability for essays read by multiple raters. The approach is motivated by generalizability theory, and variation of rater severity and rater inconsistency is considered in the presence of between-examinee variations. Illustrates methods with data from standardized educational tests. (Author/SLD)

Sensitivity theory proposes that there are wide individual differences in what motivates people with intellectual disability. The Reiss Profile MRDD is a rating scale that measures 15 fundamental motives. This study examined the internal consistency and interrater reliability of the 15 subscales as well as the validity of motivational profiles.…

Comments on the response offered by Cheng and Novick to White's initial comments on Cheng's and Cheng and Novick's previous articles. White asks if regularity information necessary for causal learning. He and Cheng and Novick agree that the causal relation is understood as a generative relation, but disagree on how this understanding comes about.…

The product of mental health counseling, unlike that of most professions, remains invisible to most people, leaving counselors vulnerable in a competitive market. The author argues that clinicians should recognize the value of, understand, and begin using outcome measures in their work. Research focusing on critical problems in psychotherapy…

H. Bosch, F. Steinkamp, and E. Boller's (see record 2006-08436-001) meta-analysis, which demonstrated (a) a small but highly significant overall effect, (b) a small-study effect, and (c) extreme heterogeneity, has provoked widely differing responses. After considering D. B. Wilson and W. R. Shadish's (see record 2006-08436-002) and D. Radin, R.…

The third edition of the "Boston Diagnostic Aphasia Examination" (Goodglass, Kaplan, and Barresi) introduced standardized procedures for coding discourse samples elicited using the well known Cookie Theft illustration. To evaluate the reliability of this discourse coding procedure, a transcribed sample was coded by 14 novice examiners…

This study presents an investigation of information literacy as defined by the ETS iSkills™ assessment and by the New Jersey Institute of Technology (NJIT) Information Literacy Scale (ILS). As two related but distinct measures, both iSkills and the ILS were used with undergraduate students at NJIT during the spring 2006 semester. Undergraduate…

To help states and districts make informed decisions about the professional development (PD) they implement to improve reading instruction, the U.S. Department of Education commissioned the Early Reading PD Interventions Study to examine the impact of two research-based PD interventions for reading instruction: (1) a content-focused teacher…

Judge-mediated examinations are defined as those for which expert evaluation (using rubrics) is required to determine correctness, completeness, and reasonability of test-taker responses. The use of multifaceted Rasch modeling has led to improvements in the reliability of scoring such examinations. The establishment of criterion-referenced…

Background: Coaches and physical educators must obtain valid data relating to the contribution of each of their players in order to assess their level of performance in team sport competition. This information must also be collected and used in real game situations to be more valid. Developed initially for a physical education class context, the…

The authors are pleased with commentators' willingness to respond to their target article's challenge to identify observable reproducible phenomena that could be widely accepted as strong scientific evidence for innate talent. In this reply, the authors have organized the ideas in the commentaries into three general categories, namely the…