This study utilizes an argument-based approach to validation to examine the implications of reliability in order to further differentiate the concepts of score and decision consistency. In a methodological example, the framework of generalizability theory was used to estimate appropriate indices of score consistency and evaluations of the…

This commentary on articles in this special issue generally agrees with the viewpoints expressed, although it argues that in some cases the authors of these articles should have expanded on certain issues. Many comments relate to the legal defensibility of the positions taken. (SLD)

A brief summary of standard setting knowledge is presented, derived from about 20 methods that utilize a judgmental review process, the approach most relevant to the standard-setting strategies proposed in this special issue. Criteria for judging effectiveness and critiques of the methods discussed in the issue are offered. (SLD)

The articles of this special issue propose two methods of deriving an initial standard and one method for determining the extent to which the standard should include compensation. Much work remains to be done on further development of the methods and the larger issues of policy regarding performance assessment. (SLD)

A performance-standard setting procedure termed judgmental policy capturing (JPC) and its application are described. A study involving 12 panelists demonstrated the feasibility of the JPC method for setting performance standards for classroom teachers seeking certification from the National Board for Professional Teaching Standards. (SLD)

The three standard-setting approaches described in this special issue are summarized and contrasted: (1) judgmental policy capturing; (2) the extended Angoff method; and (3) the dominant profile method. An integrative summary of findings is followed by recommendations for modifying the methods. (SLD)

Development of a multistage dominant profile method for setting standards on complex performance assessments is detailed. The method grew from experiences with a judgmental policy-capturing procedure and an extended Angoff method. The design of an early adolescence English language arts assessment illustrates the complexity of decisions panelists…

A brief history of developments in the assessment of the reliability of credentialing examinations is presented, and some new results are outlined that highlight the interactions among scoring, standard setting, and the reliability and validity of pass-fail decisions. Decision consistency is an important concept in evaluating credentialing…

This article provides a framework for the rest of the articles in this special issue comparing the utility of three standard-setting methods with complex performance assessments. The context of the standard setting study is described, and the methods are outlined. (SLD)

A series of 19 questions illuminates the characteristics of effective performance assessments in 3 sections: (1) the nature of assessment; (2) properties of effective tasks; and (3) making tasks meaningful and engaging. A fourth section offers practical suggestions for the construction of performance assessments and for teacher involvement. (SLD)