This three-paper dissertation systematically investigates the methodology issues related to using observational systems to observe, analyze, and interpret teachers' reformed teaching practices within the K-12 classroom context. Each paper contributes a distinct perspective to evaluate instructional practices accurately, consistently, and…

The purpose of this study was to develop a valid and reliable assessment tool to measure music educators' participation in and perceptions of their high-quality professional development (PD) experiences. A sample of 450 in-service music educators across the United States completed a 78-item rating scale embedded within 10 domains reflecting…

Recognizing the limitations of the existing frameworks and models applied to estimate the learning loss caused by COVID-19 across the world by different studies, and the urgent need for estimating such loss at the different education levels, this paper, employs the Education Value Chain Analysis ( EVCA) approach as a new paradigm, attempts to…

Background: Economists are making increasing use of measures of student achievement obtained through large-scale survey assessments such as NAEP, TIMSS, and PISA. The construction of these measures, employing plausible value (PV) methodology, is quite different from that of the more familiar test scores associated with assessments such as the SAT…

The use of multi-category scales is increasing for the monitoring of IEP goals, classroom and school rules, and Behavior Improvement Plans (BIPs). Although they require greater inference than traditional data counting, little is known about the inter-rater reliability of these scales. This simulation study examined the performance of nine…

In this paper, we provide an overview of psychometric procedures and guidelines Educational Testing Service (ETS) uses to evaluate automated essay scoring for operational use. We briefly describe the e-rater system, the procedures and criteria used to evaluate e-rater, implications for a range of potential uses of e-rater, and directions for…

Despite the wealth of accumulated research evaluating subjective wellbeing (SWB) in children and adults, the validity of scores from parallel forms of SWB measures for each age group has yet to be empirically tested. This study examines the psychometric equivalence of the child and adult forms of the personal wellbeing index (PWI) using…

This paper examines the construct measured by automated essay scoring (AES) systems. AES systems measure features of the text structure, linguistic structure, and conventional print form of essays; as such, the systems primarily measure text production skills. In the current state-of-the-art, AES provide little direct evidence about such matters…

By almost all measures of objective life quality, Indigenous Australians are disadvantaged relative to the general population. However, no measures of their Subjective Wellbeing (SWB) have been published. This paper presents the first such data, norm-referenced to the general Australian population. A total of 519 Indigenous adolescents, aged…

Automated Essay Scoring (AES) has garnered a great deal of attention from the rhetoric and composition/writing studies community since the Educational Testing Service began using e-rater[R] and the "Criterion"[R] Online Writing Evaluation Service as products in scoring writing tests, and most of the responses have been negative. While the…

This chapter addresses the psychometric challenges in assessing English language learners (ELLs) and students with disabilities (SWDs). The first section addresses some general considerations in the assessment of ELLs and SWDs, including the prevalence of ELLs and SWDs in the student population, federal and state legislation that requires the…

As the Personal Strengths Inventory (PSI) was developed to assess the strengths of adolescents, the Personal Strengths Inventory-2 (PSI-2) was developed so that children could be assessed as well. The present study examined whether the five-factor structure of the PSI would be demonstrated with the PSI-2 and whether the same measurement structure…

We investigated the effects of systematic changes in levels of treatment integrity by altering errors of commission during error-correction procedures as part of discrete-trial training. We taught 3 students with autism receptive nonsense shapes under 3 treatment integrity conditions (0%, 50%, or 100% errors of commission). Participants exhibited…

This policy brief provides principal evaluation system designers information about the technical soundness and cost (i.e., time requirements) of publicly available school climate surveys. The authors focus on the technical soundness of school climate surveys because they believe that using validated and reliable surveys as an outcomes measure can…

In this paper I explore how Margaret Beier, Lesley Miller, and Shu Wang make claims for the validity and reliability of the instrument they developed to explore the construct of "possible selves" as described in their manuscript, "Science Games and the Development of Scientific Possible Selves."