The Bookmark Standard Setting Procedure (Lewis, Mitzel, and Green, 1996) is an item-response-theory-based standard setting method that has been widely implemented by state testing programs. The primary purposes of this study were to: (1) estimate standard errors for cutscores that result from Bookmark standard settings under a generalizability…

The multivariate generalizability of the Role Functioning Scale (RFS), which serves as a replacement for the global ratings of the Social Adjustment Scale, was studied with 396 severely disturbed clients of a public mental-health bureau. Generalizability of single subscales was less than that of the combined subscales. (SLD)

Limitations of the confirmatory factor analysis conducted by H. W. Marsh to demonstrate the multidimensionality of a rating form to assess student evaluation of teaching effectiveness are discussed. A secondary analysis suggests that dimensionality is not stable across solutions and a global factor can be retrieved. (SLD)

Evidence from test results of 3,604 sixth and seventh graders is provided for the generalizability and validity of the Quantitative Understanding: Amplifying Student Achievement and Reasoning (QUASAR) Cognitive Assessment Instrument, which is designed to measure program outcomes and growth in mathematics. (SLD)

The New Standards Project conducted a pilot test of a series of performance-based assessment tasks in mathematics and English language arts at Grades 4 and 8 in the spring of 1993. This paper reports the results of a series of generalizability analyses conducted for a subset of the 1993 pilot study data in mathematics. Generalizability analyses…

This annotated bibliography contains nine items addressing assessment methodology. The titles are: "Performance and Product Evaluation"; "The Critical Incident Technique"; "Constructing Achievement Tests"; "Applying the Assessment Center Method"; "Performance Assessment in Education and Training:…

This paper discusses the theoretical scope and practical applicability of generalizability (G) theory through the principle of symmetry. Major ideas are summarized and factors hindering applications of G theory in research conducted in French-speaking Europe are presented. The principle of symmetry affirms that any factor of a design can be…

The purpose of this study was to compare the scoring of Teacher Performance Assessment Instruments (TPAI) indicators using discrete descriptors when some are considered "essential" with the scoring of these same indicators, and when no descriptors are considered essential. The two questions addressed in this study were: (1) To what…

The assessment framework underlying a science performance monitoring program is process-oriented and intended to appeal to generalizability theory for a suitable estimation paradigm. Preliminary applications are described. Results suggest that computerized question-banking, domain-sampling of questions, and generalizablity theory together provide…

School test performance is commonly summarized in terms of the percentage of students at or above a cut score (PAAC) that has been set on a test. Two approaches to estimating the standard errors for school PAACs were examined in this study: conditional standard errors and overall standard errors. The tests used were English language arts and…

Results from a performance assessment in which 68 high school students wrote essays support the use of latent variable modeling for estimating reliability, concurrent validity, and generalizability of a scoring rubric. The latent variable modeling approach overcomes the limitations of certain conventional statistical techniques in handling…

Generalizability theory is used to examine the psychometric characteristics of the Listening and Writing Tests developed by American College Testing for its Work Keys program. Results with samples of 50 suggest the desirability of a minimum number of the tests' tape-recorded messages and the use of at least 2 raters. (SLD)

A meta-analysis was conducted of 107 classroom and laboratory studies concerning the effects of expecting a recall, recognition, essay, multiple-choice, or true-false test on students' subsequent achievement. Laboratory studies did not generalize well to classrooms. In classroom studies, subjects performed better when they knew which test type to…

There is a natural tension between experimentally designed studies and case studies, which differ in that they are not concerned with the interaction of variables in the quantitative and statistical sense. This paper describes a study that was not experimentally designed, but its major findings were generalizable to the overall population of…

Selected examples using the statistical packages Statistical Package for the Social Sciences (SPSS), the Statistical Analysis System (SAS), and BMDP are presented to facilitate their use and encourage appropriate uses in: (1) a hierarchical design; (2) a confounded factorial design; and (3) variance component estimation procedures. To illustrate…