School choice programs continue to be controversial, spurring a number of researchers into evaluating them. When possible, researchers evaluate the effect of attending a school of choice using randomized designs to eliminate possible selection bias. Randomized designs are often thought of as the gold standard for research, but many circumstances…

Purpose: This AERI-NEPC eBrief, the fourth in a series titled "Understanding validity issues around the world", looks closely at issues surrounding the validity of test-based actions in educational accountability and school improvement contexts. The specific discussions here focus testing issues in the US. However, the general principles…

Contains seven articles discussing the application of meta-analysis to research on outcomes in psychotherapy. Provides a framework for understanding meta-analytic procedures and outlines major techniques. Reviews criticisms of integration of psychotherapy outcome research. Evaluates comparative therapy outcome research and discusses limitations of…

Reports that the results of a study that used both classical and naturalist experimental approaches yielded quite different linguistic pictures of the same children. (JL)

An initial research referral instrument was created and evaluated. Results suggested an effective referral instrument could more effectively capitalize upon the skill and training of a professional counselor in contributing to a more useful psychological-vocational evaluation report by a counseling psychologist. Evaluation data were used to revise…

By focusing on "appropriateness" and "adequacy" of inference and action, unified validity may be misused in rejecting valid test outcomes. The notion of levels of validity is challenged, the necessity of assumption is argued, and experience is proposed as the basis of validity. "Consequential validity" is interpreted as an optional predictive…

This article discusses evidence for functional behavioral assessment (FBA) procedures in terms of the meaning and interpretation of different forms of reliability and validity in FBA. It concludes that additional research needs to be conducted with various target behaviors across different settings to support the continued use of FBA. (Contains…

Aserts that it is unclear how the Ages and Stages Questionnaires: Social-Emotional ratings link to other critical child and family risk factors. Identifies questions regarding how well individual dimensions of social behavior are measured and concerns about the partial overlap between competent and problem behavior. Expresses concerns about the…

Despite questions about their reliability and validity, student evaluations of teaching (SETs) are a primary measure of instructor performance. The current study examines SETs, including a thorough list of potentially relevant variables. The findings suggest that how much students perceive they learned in a course is an important predictor of SET…

Threats to the validity of the Title I Evaluation and Reporting System are covered in two parts: an annotated bibliography of reports concerned with technical issues, and a discussion of threats to validity--from the reporting system in general and from each model in particular. Threats common to all three evaluation models are reported:…

Title I Evaluation models using uniform procedures and data collection for state and national comparisons are interpreted as federal endorsement of basic, common competencies in reading and mathematics. Basic competencies, content homogeneity, conversion methodology, and validity of the evaluation are discussed. (MH)

In 1987, a survey report by J. Cannell was published criticizing state reports based on national standardized testing of elementary school students that indicated that all 50 states were above the national average. This phenomenon--known as the "Lake Wobegon Effect"--is analyzed. Technical problems with the study are considered. (TJH)

The report of the National Academy of Education (NAE), "Setting Performance Standards for Student Achievement," provides an interpretation of NAE investigations into the procedures surrounding establishment of achievement levels for the National Assessment of Educational Progress (NAEP). The NAE report synthesizes the investigations of…

In September 1991, the National Assessment Governing Board (NAGB) announced standards for basic, proficient, and advanced achievement in mathematics and reported that few American students had reached these standards. Expert reviewers noted technical problems with the NAGB approach and questioned its results. In this report, the NAGB…

The National Academy of Education (NAE) Panel has drawn two major conclusions in its evaluation of the National Assessment Governing Board's (NAGB) efforts to set achievement levels for the National Assessment of Educational Progress (NAEP). They are that the Angoff procedure is fundamentally flawed for the setting of achievement levels, and that…