Work Keys provides a metric that translates skill requirements for individual jobs into levels of proficiency. It has been developed as a multifunctional program with the interactive components of Job Profiling, Instructional Support, Reporting, and Assessment. The assessment component contains applied mathematics, applied technology, teamwork,…

A new evaluation instrument was developed to measure the performance of caseworkers at a private foster care agency. It was designed to correlate directly with the caseworker's job description and to provide better data than the previous evaluation tool. Scoring forms were developed to compare the previously used evaluation instrument and the new…

This study investigated the intrajudge consistency of Angoff-based item performance estimates. The examination used was a certification examination in an emergency medicine specialty. Ten expert panelists rated the same 24 items twice during an operational standard setting study. Results indicate that the panelists were highly consistent, in terms…

This paper reports the results of an analysis of the relationship between scorer behaviors and score variability. Thirty-six essay scorers were interviewed and asked to perform a think-aloud task as they scored 24 essays. Each comment made by a scorer was coded according to its content focus (i.e. appearance, assignment, mechanics, communication,…

This paper reports on a pilot study that is designed to become an element of a larger research endeavor. The larger study will explore relationships between how teachers teach in the classroom, those teachers' perceptions of their own pedagogical practice, and the beliefs of those teachers about the roles of the student and the teacher in the…

The Ability Explorer (AE) is a newly developed self-report inventory of abilities that is appropriate for group or individual administration. There are machine-scorable and hand-scorable versions of the test, and there are two levels. Level 1 is for students from junior high to high school, and Level 2 is for high school students and adults.…

Among the measurement techniques receiving greater attention is the context-dependent item set or testlet. The context-dependent item set consists of a scenario and related test questions. This item format is generally believed to be able to tap higher level thinking. Unfortunately, this item form leads to inter-item dependence within item sets…

Modern applications of computerized adaptive testing (CAT) are typically grounded in item response theory (IRT; Lord, 1980). While the IRT foundations of adaptive testing provide a number of approaches to adaptive test scoring that may seem natural and efficient to psychometricians, these approaches may be more demanding for test takers, test…

The construct validity of content standards is addressed in terms of their representative coverage of a construct domain and their alignment with the students' cognitive level of developing expertise in the subject matter. The construct validity of performance standards is addressed in terms of the extent to which they reflect increasing levels of…

Fit analysis is widely performed in item response theory (IRT) based test development to assess the fit of individual items to the IRT model being used. The paper explores a step fit analysis procedure that is an extension of IRT-based item fit diagnostics applied to the response categories present in popular performance-based tasks. The step fit…

This paper focuses on the use of portfolio assessment in the evaluation of limited-English-proficient (LEP) students. Specific sections address the following: recent developments in performance assessment; observations regarding LEP students and assessment; student portfolios (what they are and how they can be used); what a portfolio should…

What educators around the country are finding out as they work with portfolios is reviewed, focusing on the classroom. Portfolios are purposeful, collaborative, self-reflective collections of student work generated during the process of instruction. Although they can be used in any academic discipline, portfolios are widely used in English and…

Many researchers have suggested that the main cause of item bias is the misspecification of the latent ability space. That is, items that measure multiple abilities are scored as though they are measuring a single ability. If two different groups of examinees have different underlying multidimensional ability distributions and the test items are…

Discussed are nine questions regarding standard setting issues in educational testing: (1) Should normative or content-referenced standards be used? (2) Different standard setting methods yield different results. Does this finding present a problem? (3) Assess the adequacy of the grounding of various methods of standard setting in psychological…

Asking students to select materials for tests, prepare them, score them, and analyze them is a valuable learning experience. (RC)