This study investigated the intrajudge consistency of Angoff-based item performance estimates. The examination used was a certification examination in an emergency medicine specialty. Ten expert panelists rated the same 24 items twice during an operational standard setting study. Results indicate that the panelists were highly consistent, in terms…

This book offers a descriptive account of the world of the British external examiner: who they are, what they do, how they are appointed, and what problems they face. Findings are based on a questionnaire survey of and personal interviews with examiners. The external examiner is independent of the university awarding the degree and quite…

This paper reports the results of an analysis of the relationship between scorer behaviors and score variability. Thirty-six essay scorers were interviewed and asked to perform a think-aloud task as they scored 24 essays. Each comment made by a scorer was coded according to its content focus (i.e. appearance, assignment, mechanics, communication,…

Students often have difficulty in evaluating the validity of a study. A conceptually and linguistically meaningful framework for evaluating research studies is proposed that is based on the discussion of internal and external validity of T. D. Cook and D. T. Campbell (1979). The proposal includes six key dimensions, three related to internal…

The Ability Explorer (AE) is a newly developed self-report inventory of abilities that is appropriate for group or individual administration. There are machine-scorable and hand-scorable versions of the test, and there are two levels. Level 1 is for students from junior high to high school, and Level 2 is for high school students and adults.…

It is sometimes sensible to think of the fundamental unit of test construction as being larger than an individual item. This unit, dubbed the testlet, must pass muster in the same way that items do. One criterion of a good item is the absence of differential item functioning (DIF). The item must function in the same way as all important…

Among the measurement techniques receiving greater attention is the context-dependent item set or testlet. The context-dependent item set consists of a scenario and related test questions. This item format is generally believed to be able to tap higher level thinking. Unfortunately, this item form leads to inter-item dependence within item sets…

Students at grades four and five were administered a writing assessment that was developed to correspond to the California Learning Assessment System (CLAS) writing tasks at grade four. Teachers were trained to score the CLAS-like tasks according to the rubric developed by the State for CLAS. In addition, 164 students at three schools in the…

This study evaluated the validity of using an earplug so that information is presented to only one ear of children (N=34) in Alberta, Canada, with learning disabilities as a means of improving listening comprehension. Children were asked to recall stories from the Auditory Comprehension Test (ACT) immediately after hearing the story. The children…

This study examined the relationship between scores on the Wechsler Intelligence Scale for Children-III (WISC-III) and the older Wechsler Intelligence Scale for Children-Revised (WISC-R). School psychologists in Wisconsin were asked to provide data on 300 special education re-evaluations completed during the 1992-93 academic year. Pearson product…

Test-retest reliabilities and practice effect magnitudes were considered for nine computer-simulated tasks of everyday cognition and five traditional neuropsychological tests. The nine simulated everyday memory tests were from the Memory Assessment Clinic battery as follows: (1) simple reaction time while driving; (2) divided attention (driving…

This paper presents a review of the literature on reliability in qualitative studies. Reliability is defined as the extent to which studies can be replicated, using the same methods, and getting the same results. It is the degree to which data are independent of the accidental circumstances of the research. The review includes the following three…

The multi-stage development of the Community Service Attitudes Scale (CSAS), an instrument for measuring college students' attitudes about community service, is reported. The CSAS was developed based on the helping behavior model of S. Schwartz (1977). The developed instrument was tested with two samples of 437 and 332 college students. The scales…

The analysis of small group data with hierarchical linear models is discussed, concentrating on the usefulness and reliability of such analyses using data reported by N. M. Webb (1982). Results of Webb's analyses for 96 junior high school students in small groups are compared with results obtained with random effects linear models for the analysis…

The lack of systematic psychometric information on the Suicide Opinion Questionnaire (SOQ) was addressed by investigating the factor structure and reliability of the eight-factor clinical scale model (mental illness, cry for help, right to die, religion, impulsivity, normality, aggression, and moral evil), developed for interpreting responses to…