Human raters' assessment of interpreting is a complex process. Previous researchers have mainly relied on verbal reports to examine this process. To advance our understanding, we conducted an empirical study, collecting raters' eye-movement and retrospection data in a computerised interpreting assessment in which three groups of raters (n = 35)…

Reading comprehension is often treated as a multidimensional construct. In many reading tests, items are distributed over reading process categories to represent the subskills expected to constitute comprehension. This study explores (a) the extent to which specified subskills of reading comprehension tests are conceptually conceivable to…

The construct map is a particularly good way to approach instrument development, and this author states that he was delighted to read Adam Wyse's thoughts about how to use construct maps for standard setting. For a number of popular standard-setting methods, Wyse shows how typical feedback to panelists fits within a construct map framework.…

Scholastic tests regard cognitive abilities to be domain-specific competences. However, high correlations between competences indicate either high task similarity or a dependence on common factors. The present rating study examined the validity of 12 Programme for International Student Assessment (PISA) and Third or Trends in International…

Researchers have expressed concern about "implementation" fidelity in intervention research but have not extended that concern to "assessment" fidelity, or the extent to which pre-/posttests are administered and interpreted as intended. When studying reading interventions, data gathering heavily influences the identification of…

The purpose of this article is to provide an overview of the process used to examine the inter-rater reliability of the Teacher Work Sample (TWS) Scoring Rubric involved with the senior culminating experience for teacher candidates used at a large comprehensive university. The study compared holistic and analytic scores reported by Student Teacher…

In this study we use data from the Early Childhood Longitudinal Survey third- and fifth-grade samples to investigate teacher judgments of student achievement, the extent to which they offer a similar picture of student mathematics achievement compared to standardized test scores, and whether classroom assessment practices moderate the relationship…

This research develops and tests a procedure for interpreting individuals' responses in multiscale career assessments, using the Career Decision-Making Difficulties Questionnaire (CDDQ). In Study 1, criteria for ascertaining the credibility of responses were developed, based on the judgments of 39 career-counseling experts. In Study 2, the…

The National Council for Accreditation of Teacher Education (NCATE, 2002) requires teacher education units to develop assessment systems and evaluate both the success of candidates and unit operations. Because of a stated, but misguided, fear of statistics, NCATE fails to use accepted terminology to assure the quality of institutional evaluative…

Investigated the interscorer reliability between a novice and a professional psychologist for the Minnesota Percepto-Diagnostic Test-Revised (MPDT-R), using a sample of 30 individuals. Results indicated that for three of the four MPDT-R scores there was a significant positive correlation between expert and novice scoring criteria. (JAC)

Assessed the interrater reliability of the revised Advanced Psychodiagnostic Interpretation (API) (A. Raphael and C. Golden, 1998) scoring system for the Bender Gestalt Test (L. Bender, 1938). Agreement across nine raters exceeded 90% for each of three clinical protocols, and kappa statistics indicated good interrater reliability. (SLD)

H. Bosch, F. Steinkamp, and E. Boller's (see record 2006-08436-001) meta-analysis, which demonstrated (a) a small but highly significant overall effect, (b) a small-study effect, and (c) extreme heterogeneity, has provoked widely differing responses. After considering D. B. Wilson and W. R. Shadish's (see record 2006-08436-002) and D. Radin, R.…

Examined the validity of the Adult Attachment Interview (AAI) measure by interviewing 83 mothers twice over 2 months, using different interviewers on each occasion. The results indicated that the reliability of the AAI classifications was quite high over time and across interviewers. The AAI classifications were independent of nonattachment…

Recently, "Developmental Psychology" published 2 articles on the shape bias; both rejected the authors' previous proposals about the role of attentional learning in the development of a shape bias in object name learning. A. Cimpian and E. Markman (2005; see record EJ733667) did so by arguing that the shape bias does not exist but is an…

It is incorrect to say "the test is reliable" because reliability is a function not only of the test itself, but of many factors. The present paper explains how different factors affect classical reliability estimates such as test-retest, interrater, internal consistency, and equivalent forms coefficients. Furthermore, the limits of classical test…