The same L2 speaking performance may be analysed and evaluated in very different ways by different teachers or raters. We present a new, technology-assisted research design which opens up to investigation the trajectories of convergence and divergence between raters. We tracked and recorded what different raters noticed when, whilst grading a…

In this study, generalizability theory was used to examine the extent to which (a) time-sampling methodology, (b) number of simultaneous behavior targets, and (c) individual raters influenced variance in ratings of academic engagement for an elementary-aged student. Ten graduate-student raters, with an average of 7.20 hr of previous training in…

Although scalable programs, such as online courses, have the potential to reach broad audiences, they may pose challenges to evaluating learners' knowledge and skills. Automated scoring offers a possible solution. In the current paper, we describe the process of creating and testing an automated means of scoring a validated measure of teachers'…

Early care and education programme quality is usually assessed at the classroom level. One such measure of classroom quality is the classroom assessment scoring system (CLASS). In an effort to ensure higher quality programming, the CLASS is being used to direct teacher professional development. However, there has been relatively little research on…

This study examined the reliability and structure of the Classroom Assessment Scoring System (CLASS; Pianta, R. C., K. M. La Paro, and B. K. Hamre. 2008. "Classroom Assessment Scoring System. Manual Pre-K." Baltimore, MD: Brookes) and the quality of interactional processes in a German pre-school setting, drawing on a sample of 390…

Evidence from a handful of large-scale studies suggests that although observers can be trained to score reliably using observation protocols, there are concerns related to initial training and calibration activities designed to keep observers scoring accurately over time (e.g., Bell, et al, 2012; BMGF, 2012). Studies offer little insight into how…

The current study examined the stability of temperament over time. Observers and mothers rated child behavior at eight timepoints across three assessment waves (8, 15, and 30 months of age). Internal consistency reliability of aggregates of the eight observer reports and eight mother reports were high. When considering single timepoint…

The purpose of this study was to examine the use of an emerging technology called augmented reality to teach science vocabulary words to college students with intellectual disability and autism spectrum disorders. One student with autism and three students with an intellectual disability participated in a multiple probe across behaviors (i.e.,…

States and districts can use this tool to create their own plans for continual improvement of an observation system, no matter where they are in their implementation. Included are action steps to improve observation rubrics, observer training, observer assessment, and monitoring. A planning process is described to assess current status, determine…

Shaping functional vocal language is difficult when an individual has not yet acquired an echoic repertoire and does not emit sufficient phonemes (i.e., speech sounds) for shaping. Few studies have evaluated interventions to increase the frequency and breadth of phonemes. The current study extended Esch, Esch, and Love (2009) by evaluating the…

There is a growing consensus that teacher evaluation in the United States is fundamentally broken. Few would argue that a system that tells 98 percent of teachers they are "satisfactory" benefits anyone--including teachers. The nation's collective failure to invest in high-quality professional feedback to teachers is inconsistent with…

Research has long been clear that teachers matter more to student learning than any other in-school factor. Improving the quality of teaching is critical to student success. Yet only recently have many states and districts begun to take seriously the importance of evaluating teacher performance and providing teachers with the feedback they need to…