Nearly all of the research on standard setting focuses on different standard setting methods rather than the interaction of group members and the instructions given to group members. This study explored the effect of deliberation style and the requirement to reach consensus on the passing score, on rater satisfaction, and on postdecision…

Experimenter bias effects were experimentally manipulated in a sample of 97 school psychologists scoring 3 subscales (Similarity, Vocabulary, and Comprehension) of the Wechsler Intelligence Scale for Children-Third Edition (WISC-III). First year (n=29), interns (n=42), and experienced (n=26) psychologists were randomly assigned to either a bias or…

A Web-based survey of 25 experts in testing theory and large-scale assessment examined the utility of out-of-level testing for making decisions about students and schools. Survey respondents were given a series of scenarios and asked to judge the degree to which out-of-level testing would affect the reliability and validity of test scores within…

The paper discusses the key criteria of good language tests: practicality, validity, and reliability.

The "Rating Ethical Content Scale" ("RECS") judges the content of stories for positive content, based on the Four Process model of ethical behavior: ethical sensitivity, ethical judgment, ethical focus and ethical action (Rest, 1983; Narvaez, & Rest, 1995). For example, a story with Ethical Sensitivity has evidence of…

NCATE (2002) requires the measurement of knowledge, skills, and dispositions as part of its accreditation requirements for teacher education programs (Standard 1) and the use of unit assessment systems to aggregate and analyse data with a view toward program improvement (Standard 2). Data must indicate that candidates meet professional, state, and…

The concept of self-directed learning has become increasingly important in educational and work organizations as a result of trends that require learners to become more responsible for their own learning to remain highly skilled and knowledgeable in a competitive marketplace. To assess self-directedness in the Korean context, a relatively new…

This paper illustrates a method to study rater severity across exam administrations. A multi-facet Rasch model defined the ratings as being dominated by four facets: examinee ability, rater severity, project difficulty, and task difficulty. Ten years of data from administrations of a histotechnology performance assessment were pooled and analyzed…

This report provides empirical results of attempts to achieve consistency of estimates between two National Center for Education Statistics (NCES) surveys. These surveys are the 1991- 92 Private School Survey (PSS) and the Private School Component of the 1990-91 Schools and Staffing Survey (SASS). Consistency was sought in the numbers of schools,…

The short-term (45-day) stability of the Adjustment Scales for Children and Adolescents (P. McDermott, N. Marston, and D. Stott, 1993) was studied with 51 first and fifth graders, seven of whom were classified as "exceptional/disabled." Significant test-retest reliability coefficients were obtained, and mean differences from test to…

Bootstrap analysis, both for nonparametric statistical inference and for describing sample results stability and replicability, has been gaining prominence among quantitative researchers in educational and psychological research. Procedurally, however, it is often quite a challenge for quantitative researchers to implement bootstrap analysis in…

This paper discusses the use of two theories of item analysis and test construction, their strengths and weaknesses, and applications to the design of the Hawaii State Test of Essential Competencies (HSTEC). Traditional analyses of the data collected from the HSTEC field test were viewed from the perspectives of item difficulty levels and item…

This empirical study used data from the Reading: Basic Understanding section of the New Standards English Language Arts Examination. Data were collected for 3,200 high school students randomly selected from those who took the examination. The resulting sample had 16 raters who scored 200 students each, with each student rated by only 1 rater. The…

An exploratory reliability and validity study was conducted of a relatively new response scale developed in the marketing field. Unlike many Likert-type scales, the "unbounded write-in" scale is claimed to produce distributions that more closely approximate normal distributions. This type of scale has been used in large-scale marketing studies.…

This paper reports on the development of an empirically valid and reliable assessment instrument that identifies areas of need in violence prevention skills within the student population. The completed instrument should allow a school district to choose a curriculum that aligns with their identified need at each developmentally-appropriate level…