ERIC Number: ED628198
Record Type: Non-Journal
Publication Date: 2021
Pages: 8
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Quantified Qualitative Analysis: Rubric Development and Inter-Rater Reliability as Iterative Design
McCarthy, Kathryn S.; Magliano, Joseph P.; Snyder, Jacob O.; Kenney, Elizabeth A.; Newton, Natalie N.; Perret, Cecile A.; Knezevic, Melanie; Allen, Laura K.; McNamara, Danielle S.
Grantee Submission, Paper presented at ICLS (2021)
The objective in the current paper is to examine the processes of how our research team negotiated meaning using an iterative design approach as we established, developed, and refined a rubric to capture comprehension processes and strategies evident in students' verbal protocols. The overarching project comprises multiple data sets, multiple scientists across (distant) institutions, and multiple teams of discourse analysts who are tasked with scoring over 20,000 verbal protocols (i.e., think aloud, self-explanation) collected in studies conducted in the last decade. Here, we describe the iterative modifications, negotiations, and realizations while coding our first subset comprising 7,559 individual verbal protocols. Drawing upon work in design research, we describe a process through which the research team has negotiated meaning around theory-driven codes and how this work has influenced our own ways of conceptualizing comprehension research, theory, and practice. [This paper was published in: "ICLS 2021 Proceedings," ISLS, 2021, pp. 139-146.]
Publication Type: Reports - Evaluative; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: N/A
IES Funded: Yes
Grant or Contract Numbers: R305A190063; R305A180144
Author Affiliations: N/A