A common interpretation of achievement test results is that they provide measures of achievement that are much like other measures we commonly use for height, weight, or the cost of goods. In a limited sense, such interpretations are correct, but some nuances of these interpretations have important implications for the use of achievement test…

We study the properties of two specification tests that have been applied to a variety of estimators in the context of value-added measures (VAMs) of teacher and school quality: the Hausman test for choosing between random and fixed effects and a test for feedback (sometimes called a "falsification test"). We discuss theoretical…

Accurate indicators of educational effectiveness are needed to advance national policy goals of raising student achievement and closing social/cultural based achievement gaps. If constructed and used appropriately, such indicators for both program evaluation and the evaluation of teacher and school performance could have a transformative effect on…

Defining what teachers need to know to teach algebra successfully is important for informing teacher preparation and professional development efforts. Based on prior research, analysis of video, interviews with teachers, and analysis of textbooks, we define categories of knowledge and practices of teaching for understanding and assessing teachers'…

The purpose of this validity study was to improve measurement of coaching efficacy, an important variable in models of coaching effectiveness. A revised version of the coaching efficacy scale (CES) was developed for head coaches of high school teams (CES II-HST). Data were collected from head coaches of 14 relevant high school sports (N = 799).…

A review of the literature was made of the various procedures for sequencing instructional units in individualized instruction programs. Procedures for validating sequences of instructional units, or learning hierarchies, were found to fall into two general categories. One category included procedures based on coefficients of dependence, while the…

It is understandable that parents, policy makers, educators, etc. want to know how schools are functioning. Extensive resources are expended on the educational enterprise and it is only reasonable that the impact of those resources be determined. However, determining the amount of change in students' skills and knowledge is not easy. Further,…

It is demonstrated, theoretically and empirically, that item sets can be selected that meet the unidimensionality assumption of most item response theory models, even though they require more than one ability for a correct response. A method for identifying such item sets for test development purposes is presented. (SLD)

This study provided initial validity evidence for multidimensional measures of coaching competency derived from the Coaching Competency Scale (CCS). Data were collected from intercollegiate men's (n = 8) and women's (n = 13) soccer and women's ice hockey teams (n = 11). The total number of athletes was 585. Within teams, a multidimensional…

Item response theory (IRT) has proven to be a very powerful and useful measurement tool. However, most of the IRT models that have been proposed, and all of the models commonly used, require the assumption of unidimensionality, which prevents their application to a wide range of tests. The few models that have been proposed for use with…

The concept of item discrimination is generalized to the case in which more than one ability is required to determine the correct response to an item, using the conceptual framework of item response theory and the definition of multidimensional item difficulty previously developed by M. Reckase (1985). (SLD)

The purpose of this paper is to present a generalization of the concept of item difficulty to test items that measure more than one dimension. Three common definitions of item difficulty were considered: the proportion of correct responses for a group of individuals; the probability of a correct response to an item for a specific person; and the…

A live tailored testing study was conducted to compare the results of using either the one-parameter logistic model or the three-parameter logistic model to measure the performance of college students on multiple choice vocabulary items. The results of the study showed the three-parameter tailored testing procedure to be superior to the…

Summarizes the current state of the art in test construction and contrasts it with previous conceptual models, some of which are wrong or misleading. New methodologies for item selection and review are presented, with current thinking on the specification of technical characteristics of tests. (Author/SLD)

Standard setting is a fairly widespread activity in educational and psychological measurement, but there is no formal psychometric theory to guide the development of standard setting methodology. This paper presents a conceptual framework for such a psychometric theory and uses the conceptual framework to analyze a number of methods for setting…