In the last twenty-five years, the discussion surrounding validity evidence has shifted both in language and scope, from the work of Messick and Kane to the updated Standards for Educational and Psychological Testing. However, these discussions haven't necessarily focused on best practices for different types of instruments or assessments, taking…

Many colleges face the challenge of recruiting examinees for institutional assessment programs. Unfortunately, students recognize these programs as low-stakes, and therefore may not have high performance motivation. This study examined the interrelationships between students' preferred incentives for taking the test, their performance motivation,…

Describes an approach to addressing issues of validity of inferences and the extrapolation of inferences to target domains beyond the assessment for alternative assessments. Makes the case that in both language testing and educational assessment the roles of language and content knowledge must be considered, and that the design and development of…

Authentic and direct assessments of performances and products are conceptualized in terms of multiple distinctions having implications for validation. These include contrasts between performances and products, between assessment of performance per se and performance assessment of competence or other constructs, between structured and unstructured…

While no one instrument can accurately evaluate or assess every college or university debate program, two separate frameworks can be established based in forensics and assessment literature. The first framework is a descriptive/analytic/evaluative framework for a debate program based on the philosophies of the several national conferences on…

This paper reports the results of an analysis of the relationship between scorer behaviors and score variability. Thirty-six essay scorers were interviewed and asked to perform a think-aloud task as they scored 24 essays. Each comment made by a scorer was coded according to its content focus (i.e. appearance, assignment, mechanics, communication,…

Recently there has been legislation aimed at aligning education and industry interests to better meet the needs of students, workers, and employers. After the passage of the Perkins Vocational and Applied Technology Act in 1990 and its reauthorization in 1992, the U.S. Departments of Labor and Education jointly initiated funding for 22 projects to…

Some of the issues regarding the estimation of reliability for performance assessments are explored, and a methodology is suggested for determination of reliability. In performance assessment, the magnitude of the reliability coefficient may be less than that obtained for a standardized test, since part of the potential use of performance…

An example application of portfolio assessment was developed and the model and estimates of reliability derived from the literature were then used to estimate the characteristics of an operational large-scale portfolio assessment program. Costs were estimated to put results in a realistic context. (SLD)

A study was conducted to begin a process of validating hypothesized causes of local item dependence (LID) in large-scale performance assessments. Data for the study are item level scores from 26 science tasks from the 1993 edition of the Maryland School Performance Assessment Program. Causes of high LID were hypothesized from studies by Ferrara et…

In contrast to multiple choice, alternative modes of assessment afford varying degrees of openness in the allowable responses. Prominent among the alternatives is the assessment of performance, sometimes in its own right where the issue is the quality of the particular performance per se, but more often as a vehicle for the assessment of…

In the fall of 1992, the National Association of Test Directors (NATD) surveyed its members on their involvement in the development of performance assessments and scoring rubrics. The extent to which members have developed performance assessments is reported, along with how they went about it, the advice they would offer others who are developing…

Generalizability of performance-based assessment scores across raters and tasks is examined, focusing on implications of generalizability analyses for specific uses and interpretations of assessment results. Although it seems probable that assessment conditions, task characteristics, and interactions with instructional experiences affect the…

This study demonstrates the advantages of using a constrained optimization algorithm to explore the optimal number of prompts, modes of discourse, and raters for achieving an acceptable level of reliability during a direct writing assessment. Writing samples elicited from 50 college students were rated by 3 graduate students and the scores…

Educational assessment concerns inference about student knowledge, skills, and accomplishments. Test theory has evolved in part to address questions of weight, coverage, and import of data. Resulting concepts and techniques can be viewed as applications of more general principles for inference in the presence of uncertainty. (SLD)