This Digest, intended as an instructional aid for beginning research students and a refresher for researchers in the field, identifies key factors that play a critical role in determining the credibility that should be given to a specific research study. The needs for empirical research, randomization and control, and significance testing are…

Matrix sampling of test items, the division of a set of items into different versions of a test form, is used by several large-scale testing programs. This Digest discusses nine categories of costs associated with matrix sampling. These categories are: (1) development costs; (2) materials costs; (3) administration costs; (4) educational costs; (5)…

Test validity refers to the degree to which the inferences based on test scores are meaningful, useful, and appropriate. Thus, test validity is a characteristic of a test when it is administered to a particular population. This article introduces the modern concepts of validity advanced by S. Messick (1989, 1996, 1996). Traditionally, the means of…

Research on student evaluation of college teachers' performance is briefly summarized. Lawrence M. Aleamoni offers four arguments in favor of student ratings: (1) students are the main source of information about the educational environment; (2) students are the most logical evaluators of student satisfaction and effectiveness of course elements;…

The constructs used to evaluate research quality--valid, objective, reliable, generalizable, randomized, accurate, authentic--are not value-free. They all require human judgment, which is affected inevitably by cultural norms and values. In the case of research involving American Indians and Alaska Natives, assessments of research quality must be…

Student ratings of instruction are widely used as a basis for personnel decisions and faculty development recommendations. This digest discusses concerns about the validity of student ratings and presents a case for their use in teacher evaluation. There are several strong arguments for using student ratings to evaluate teachers. Students are in a…

Abuses and misuses of statistics are frequent. This digest attempts to warn against these in three broad classes of pitfalls: sources of bias, errors of methodology, and misinterpretation of results. Sources of bias are conditions or circumstances that affect the external validity of statistical results. In order for a researcher to make…

Several common sources of error in assessment that depends on the use of judges are identified, and ways to reduce the impact of rating errors are examined. Numerous threats to the validity of scores based on ratings exist. These threats include: (1) the halo effect; (2) stereotyping; (3) perception differences; (4) leniency/stringency error; and…

Despite a history of conflicting research on its reliability and validity, student evaluation of faculty (SEF) has typically not been viewed as an infringement on academic freedom; it has generally been taken for granted that SEF is appropriate and necessary. However, informal and reasoned analyses of the issue indicate that because SEF is used…

As career development becomes established in Canadian secondary schools, the pressure increases to use performance assessments to demonstrate both the effectiveness of programs and the soundness of instructional decisions. This digest examines issues surrounding performance assessments of career development programs. Performance assessments can be…