Many methods are used to measure interrater reliability for studies where each target receives ratings by a different set of judges. The purpose of this study is to explore the use of hierarchical modeling for estimating interrater reliability using the intraclass correlation coefficient. This study provides a description of how the ICC can be…

A latent variable modeling approach for scale reliability evaluation in heterogeneous populations is discussed. The method can be used for point and interval estimation of reliability of multicomponent measuring instruments in populations representing mixtures of an unknown number of latent classes or subpopulations. The procedure is helpful also…

Internationally, the political appetite for educational measurement capable of capturing a metric of value for money and effectiveness has momentum. While most would agree with the need to assess costs relevant to quality to help support better governmental policy decisions about public spending, poorly understood measurement comes with unintended…

This article proposes that sampling design effects have potentially huge unrecognized impacts on the results reported by large-scale district and state assessments in the United States. When design effects are unrecognized and unaccounted for they lead to underestimating the sampling error in item and test statistics. Underestimating the sampling…

A jack knife is a pocket knife that is put to many tasks, because it's ready to hand. Often there could be a better tool for the job, such as a screwdriver, a scraper, or a can-opener, but these are not usually pocket items. In statistical terms, the expression implies making do with what's available. Another simile, of an extreme situation, is…

No one has a bigger stake in teaching effectiveness than students. Nor are there any better experts on how teaching is experienced by its intended beneficiaries. Only recently have many policymakers and practitioners come to recognize that--when asked the right questions, in the right ways--students can be an important source of information on the…

The "World Report on Children and Violence", (Pinheiro, 2006) was produced at the request of the UN Secretary General and the UN General Assembly. This report recommended improvement in research on child abuse. ISPCAN representatives took this charge and developed 3 new instruments. We describe this background and introduce three new measures…

Multiple matrix sampling is a sampling design in which both test items and examinees are randomly sampled from their respective populations. This study was designed to develop and assess a method for computing an estimate of a correlation coefficient when a multiple matrix sampling design is used. The examinee populations included 212 third-grade…

Statistically based risk assessment devices are widely used in criminal justice settings. Their promise remains largely unfulfilled, however, because assumptions and premises requisite to their development and application are routinely ignored and/or violated. This article provides a brief review of the most salient of these assumptions and…

Analysis of spontaneous language samples of 10 children (ages 8-9) with specific language impairments found that interviews were a reliable, valid, and efficient assessment context, eliciting the same profile of behaviors as a freeplay context without altering diagnostic classifications. (Author/JDD)

With the increasing globalization of human resources practices, a commensurate increase in demand has occurred for multi-language ("global") personality norms for use in selection and development efforts. The combination of data from multiple translations of a personality assessment into a single norm engenders error from multiple sources. This…

This symposium deals with recent issues in the development of the National Assessment model. General goals are outlined and the following topics are discussed: "Objectives and Exercises" (Jack C. Merwin); "Sampling" (A. Finkner); and "Data Analysis" (John Milholland). (CK) Aspect of National Assessment (NAEP) dealt…

Discusses the evaluation of second and foreign language programs, focusing on whether such evaluations should be summative or formative; use outside experts or program staff; emphasize qualitative or quantitative data; and concentrate on the process or the product. An annotated bibliography discusses six important works in the field. (78…

This paper examines issues in designing post-school follow-up studies in special education. The examination focuses on survey research techniques, which are widely used in the investigation of post-school adjustment of former students with handicaps. In special education, survey research studies are used commonly to address many important…

Measuring attainment of the program's objectives and describing the program's implementation are listed as two of the evaluator's major responsibilities. The description should include an explanation of the context in which the program was initiated, as well as the component materials and activities. This booklet has three purposes: (1) suggesting…