Adoption of new instructional standards in science demands high-quality information about classroom practice. Teacher portfolios can be used to assess instructional practice and support teacher self-reflection anchored in authentic evidence from classrooms. This study investigated a new type of electronic portfolio tool that allows efficient…

Accurate and rapid identification of students displaying behavioral problems requires instrumentation that is user friendly and reliable. The purpose of the study was to evaluate a multi-item direct behavior rating scale called the Direct Behavior Rating-Multiple Item Scale (DBR-MIS) for disruptive behavior to determine the number of…

This study utilizes an argument-based approach to validation to examine the implications of reliability in order to further differentiate the concepts of score and decision consistency. In a methodological example, the framework of generalizability theory was used to estimate appropriate indices of score consistency and evaluations of the…

Though the Chemistry Self-Concept Inventory (CSCI) was developed to study one aspect of the affective domain in college chemistry students, the instrument on which it was based, the Self-Description Questionnaire III, was developed for use with late adolescents. As such, we explored data generated from administering the CSCI to high school…

We developed and evaluated an Oceanography Concept Inventory (OCI), which used a mixed-methods approach to test student achievement of 11 learning goals for an introductory-level oceanography course. The OCI was designed with expert input, grounded in research on student (mis)conceptions, written with minimal jargon, tested on 464 students, and…

As a property of test scores, reliability/dependability constitutes an important psychometric consideration, and it underpins the validity of measurement results. A review of interpreter certification performance tests (ICPTs) reveals that (a) although reliability/dependability checking has been recognized as an important concern, its theoretical…

The current study served to extend previous research on scaling construction of Direct Behavior Rating (DBR) in order to explore the potential flexibility of DBR to fit various intervention contexts. One hundred ninety-eight undergraduate students viewed the same classroom footage but rated student behavior using one of eight randomly assigned…

Policymakers and school administrators have embraced value-added models of teacher effectiveness as tools for educational improvement. Teacher value-added estimates may be viewed as complicated scores of a certain kind. This suggests using a test validation model to examine their reliability and validity. Validation begins with an interpretive…

Background: Interest in the role of metacognition has been steadily rising in most forms of education. This study focuses on the construction of a questionnaire for measuring metacognitive knowledge, metacognitive regulation and metacognitive responsiveness among students in higher education and the subsequent process of testing to determine its…

Meta-assessment, or the assessment of assessment, can provide meaningful information about the trustworthiness of an academic program's assessment results (Bresciani, Gardner, & Hickmott, 2009; Palomba & Banta, 1999; Suskie, 2009). Many institutions conduct meta-assessments for their academic programs (Fulcher, Swain, & Orem, 2012),…

Observational methods are increasingly being used in classrooms to evaluate the quality of teaching. Operational procedures for observing teachers are somewhat arbitrary in existing measures and vary across different instruments. To study the effect of different observation procedures on score reliability and validity, we conducted an experimental…

Phonics was identified as one of the critical components in reading development by the National Reading Panel. Over time, research has repeatedly identified phonics as important to early reading development. Given the compelling evidence supporting the teaching of phonics in early reading, it is critical to make sure that instructional decisions…

In the first part of this article, the use of Generalizability (G) theory in examining the dependability of concept map assessment scores and designing a concept map assessment for a particular practical application is discussed. In the second part, the application of G theory is demonstrated by comparing the technical qualities of two frequently…

Peer and self-ratings have been strongly recommended as the means to adjust individual contributions to group work. To evaluate the quality of student ratings, previous research has primarily explored the validity of these ratings, as indicated by the degree of agreement between student and teacher ratings. This research describes a…

Self- and peer assessments are becoming more popular in classrooms, but there are few data on the reliability and validity of such assessments performed by school children. Because these factors are greatly affected by the number of raters, we conducted two studies to determine the rating behaviours of teenagers in self- and peer assessments, and…