Cross-validation is a common statistical procedure applied to problems that are otherwise computationally intractable. It is often employed to assess the effectiveness of prediction procedures. In this report, cross-validation is discussed in terms of "U"-statistics. This approach permits consideration of the statistical properties of…

Quantitative fairness procedures have been developed and modified by ETS staff over the past several decades. ETS has been a leader in fairness assessment, and its efforts are reviewed in this report. The first section deals with differential prediction and differential validity procedures that examine whether test scores predict a criterion, such…

Scores from the "GRE"® revised General Test provide important information regarding the verbal and quantitative reasoning abilities and analytical writing skills of applicants to graduate programs. The validity and utility of these scores depend upon the degree to which the scores predict success in graduate and business school in…

The "ETS"® Proficiency Profile (EPP), a college-level assessment, has been widely used to evaluate general education student learning outcomes (SLOs) in college. The purpose of this study was to investigate validity evidence for the EPP by evaluating the relationship with outcomes such as student retention, cumulative grade point average…

Predicting and analyzing multimodal dialog user experience (UX) metrics, such as overall call experience, caller engagement, and latency, among other metrics, in an ongoing manner is important for evaluating such systems. We investigate automated prediction of multiple such metrics collected from crowdsourced interactions with an open-source,…

A little-known theorem, a generalization of Pythagoras's theorem, due to Pappus, is used to present a geometric explanation of various definitions of the contribution of component tests to their composite. I show that an unambiguous definition of the unique contribution of a component to the composite score variance is present if and only if the…

This survey study investigated how graduate school admissions committees perceive and use the "GRE"® General Test and "GRE"® Subject Tests after the launch of the "GRE"® revised General Test in August 2011. These perceptions and uses impact the validity of the tests. Prior research about the perceptions and uses of…

Proficiency with written communication (WC) is critical for success in college and careers. As a result, institutions face a growing challenge to accurately evaluate their students' writing skills to obtain data that can support demands of accreditation, accountability, or curricular improvement. Many current standardized measures, however, lack…

The report is the first systematic evaluation of the sentence equivalence item type introduced by the "GRE"® revised General Test. We adopt a validity framework to guide our investigation based on Kane's approach to validation whereby a hierarchy of inferences that should be documented to support score meaning and interpretation is…

For a testing program with frequent administrations, it is important to understand and monitor the stability and fluctuation of test performance across administrations. Different methods have been proposed for this purpose. This study explored the potential of using multilevel analysis to understand and monitor examinees' test performance across…

In most large-scale programs of tests that aid in making high-stakes decisions, such as the "TOEIC"® family of products and service, it is not unusual for a significant portion of test takers to retake the test at multiple times.The study reported here used multilevel growth modeling to explore the score change patterns of nearly 20,000…

In this research, we investigated the feasibility of implementing the "e-rater"® scoring engine as a check score in place of all-human scoring for the "Graduate Record Examinations"® ("GRE"®) revised General Test (rGRE) Analytical Writing measure. This report provides the scientific basis for the use of e-rater as a…

International institutions have been increasingly using the "GRE"® revised General Test to admit students to graduate programs.However, little is known about how scores from the GRE revised General Test are used in the admission process outside of the United States and their validity in predicting graduate students' performance (e.g.,…

One concern when repurposing a test to a new population is whether the test is measuring the same construct in a valid and reliable way that is comparable to the intended population. Following the guidelines of the International Test Commission and the ETS Standards for Quality and Fairness, this study was designed to collect evidence in support…

Language test preparation has often been studied within the consequential validity framework in relation to ethics, equity, fairness, and washback of assessment. The use of independent and integrated speaking tasks in the "TOEFL iBT"® test represents a significant development and innovation in assessing speaking ability in academic…