RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two…

In this article, we examine whether cross-national studies disclose enough information for independent researchers to evaluate the validity and reliability of the findings (evaluation transparency) or to perform a direct replication (replicability transparency). The first contribution is theoretical. We develop a heuristic theoretical model…

This report presents a review of the statistical practices of 30 journals representative of the second language field. A review of 150 articles showed a number of prevalent statistical violations including incomplete reporting of reliability, validity, non-significant results, effect sizes, and assumption checks as well as making inferences from…

Debate is occurring about which result interpretation aides focused on examining the experimental effect should be used in single-subject experimental research. In this study, we examined seven nonoverlap methods and compared results using each method to judgments of two visual analysts. The data sources for the present study were 36 studies…

Measurement of affective states in everyday life is of fundamental importance in many types of quality of life, health, and psychological research. Ecological momentary assessment (EMA) is the recognized method of choice, but the respondent burden can be high. The day reconstruction method (DRM) was developed by Kahneman and colleagues ("Science,"…

A study was conducted at Los Angeles Pierce College (LAPC) to test the relative reliability of small-sample telephone interviewing and large-sample mailings with postcard and telephone reminders for student follow-up. The study involved conducting a telephone survey of 245 former students who had enrolled at LAPC in four program areas, and a mail…

Special problems confront the researcher measuring the attitudes of young children who cannot readily make cognitive judgments associated with conventional psychometric scales, such as Likert scales. One measurement strategy, called the method of paired comparisons, can overcome these difficulties. The method is explained in this paper. A small…

The problem of the comparability of change scores is investigated. Change quotients and residual change scores are evaluated as alternative approaches and methods for estimating the true change and true score residual, the reliability of change scores and residuals, and procedures for constructing confidence intervals for residuals are explored.…

During round 16 of the National Longitudinal Survey of Youth (NLSY), 900 NLSY sample members were randomly assigned to be interviewed about the period since their round 14 interview. Their responses were compared to those of approximately 8,000 NLSY sample members who were assigned to be interviewed about the 1-year period since their round 15…

A study compared the nature and frequency of faults and errors in reference list entries and direct quotes selected from 78 articles in 8 issues of the "American Educational Research Journal" (AERJ) with authors sampled from 10 previously studied journals. All departures from the original (additions, omissions, or changes) were labeled as either a…

To evaluate the accuracy of the 1970 census' occupational classification system, a comparison was made between that year's responses to census questions and the replies by the same population group to the 1972 Postcensal Manpower Survey (PMS). Each of these questionnaires relied on a different methodology; in the PMS the respondent classified…

It is argued that research testing the reliability and validity of college and university reputational studies should be conducted, a concern not raised by consumers and publishers. Two studies of "America's Best Colleges," published by "U.S. News and World Report," raise questions about the guide's validity and suggest that…

Debate grows over the validity and reliability of college and university rankings, which judge rather than describe institutions, as the public's willingness to accept them increases. While many institutional characteristics can be measured, measurement methods need improvement. One solution is to publish the peer studies that the institutions…

A study was conducted by New York's Broome Community College (BCC) to compare student-reported parental income with income reported by the students' parents on their 1982 tax records. Student reports of parental income were obtained from a survey administered to students entering BCC in fall 1983. The college had parents' federal income tax…

The purpose of this study was to improve both statistical and judgmental methods for detecting potentially biased test items in an attempt to examine the agreement between the results obtained with these methods. If greater agreement between methods can be achieved, test items can be more effectively screened using judgmental methods prior to…