ERIC Number: ED388725
Record Type: Non-Journal
Publication Date: 1995
Pages: 98
Abstractor: N/A
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Monitoring and Improving a Portfolio Assessment System.
Myford, Carol M.; Mislevy, Robert J.
Establishing and refining a framework for performance assessment is especially difficult in large-scale settings that can involve hundreds of judges and thousands of students. This presentation advocates the interactive use of two complementary analytic perspectives and illustrates the approach in the context of the College Entrance Examination Board's Advanced Placement (AP) Studio Art portfolio assessment. The "naturalistic" component of the project involved in-depth discussions with judges about 18 portfolios from the 1992 assessment that received discrepant ratings. Since it is impossible to hold such discussions for each of the individual ratings produced in the assessment, summary results for each, in the form of numerical ratings, provided the data for the "statistical" component. J. M. Linacre's (1989) FACETS model was used to summarize overall effect patterns, quantify the evidence associated with these effects, and highlight rating profiles and judge/portfolio combinations that are unusual in light of typical patterns. This focused attention where it was apt to be most useful in improving the process. By using both perspectives, one can better communicate the meaning, value, and quality of the assessment. Two appendixes contain the AP Studio Art Reader Survey and the judge interview protocol. (Contains 11 tables, 3 figures, and 26 references.) (SLD)
Descriptors: Advanced Placement, Art Products, Educational Assessment, Educational Improvement, Evaluation Methods, Interrater Reliability, Models, Naturalistic Observation, Performance Based Assessment, Portfolio Assessment, Portfolios (Background Materials), Profiles, Test Construction, Testing Programs
Center for Performance Assessment, Educational Testing Service, Mail Stop 11-P, Rosedale Road, Princeton, NJ 08541-0001.
Publication Type: Reports - Evaluative; Tests/Questionnaires
Education Level: N/A
Audience: N/A
Language: English
Sponsor: College Entrance Examination Board, Princeton, NJ.; Office of Educational Research and Improvement (ED), Washington, DC.
Authoring Institution: National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, CA.; Educational Testing Service, Princeton, NJ. Center for Performance Assessment.
Identifiers - Assessments and Surveys: Advanced Placement Examinations (CEEB)
Grant or Contract Numbers: N/A
Author Affiliations: N/A