ERIC Number: ED677785
Record Type: Non-Journal
Publication Date: 2025-Oct-11
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
Learning from Null Results -- The Importance of Research Partnerships in Building Knowledge
Laura Holian; Ben Gold; Catharine Warner-Griffin; Vicki Kirk
Society for Research on Educational Effectiveness
Background/Context: The Education Innovation and Research (EIR) grant program provides funding for creating and implementing innovative approaches to improving student achievement while rigorously evaluating the approach. The STEM.LD program was funded by a 2020 mid-phase EIR grant. It was designed in response to a growing need for individuals with STEM content knowledge and skills and the view that students need to become interested and engaged in math and science in middle school. The design of the program built on lessons learned from a prior i3 funded project (Mokher, Lee, and Sun, 2019) in the same region that provided expanded access to rigorous coursework, and from TNTP's Opportunity Myth framework. STEM.LD was designed to incorporate in-person and virtual STEM-rich learning environments in rural northeast Tennessee for students to accelerate learning and exposure to STEM subjects while exploring their interests in these fields. Purpose/Objective/Research Question: To address a need in northeastern Tennessee for students to receive quality education and training in STEM fields and pursue in-demand careers, the Niswonger Foundation created the STEM.LD program. The STEM.LD program provided teachers with the resources to design student-centered learning environments and pathways that encourage interest and engagement in STEM content. By building educator STEM knowledge, skill, and pedagogy through professional development; increasing access to and awareness of high-quality OST activities; and providing opportunities for students to increase their academic readiness, the program was hypothesized to increase middle school student science and math performance on state tests. It answered: What is the effect of two years of attending a middle school supported by the STEM.LD program on 7th & 8th grade students' mathematics and science achievement compared with 7th and 8th grade students in the business as usual condition? Setting: The program takes place in public middle school buildings across 14 school districts in predominantly rural, northeast Tennessee. Teachers participated in online professional development modules and implemented instructional practices in their classrooms. Students had access to the courses offered at their school or online. Most out-of-school time activities occurred on school grounds after the school day or on weekends. Some OST activities took place at partner community college locations. Population/Participants/Subjects: Overall, 40 schools in rural, northeast Tennessee volunteered to be part of the study. To be eligible to participate, the schools had to have at least two of the three middle grade levels, be willing to participate in data collection for at least two years, and be willing to be randomly assigned to treatment or control groups. 28 schools volunteered to participate starting in 2021-22, and an additional 12 schools starting in 2022-23. They were randomized into receiving the STEM.LD intervention or serving as a comparison group with the option for delayed implementation. The study included all students in grades 6-7 at the participating schools at the start of treatment. Students missing the outcome data, baseline data, or student characteristics data necessary to address a research question were excluded from the analytic sample for that question. There were 6,560 students in the math sample (3,655 in the treatment group and 2,905 in the control group) and 6,715 in the science sample (3,730 in the treatment group and 2,985 in the control group). Intervention/Program/Practice: STEM.LD provides teachers with the resources to design student-centered learning ecosystems and pathways that encourage interest and engagement in STEM content. STEM.LD is a two-year, school-wide program with four major components: 1) 6 hours of asynchronous professional learning for math and science teachers and leaders on student engagement in instruction, plus additional half day PD sessions; 2) deep dive professional development for teachers around topics such as standards-based math and science training, cybersecurity, synthetic biology, applied measurement, epidemiology, computational statistics, GIS, technology and design thinking, and/or aerospace engineering; 3) learning environments with rich STEM resources; and 4) access to STEM coursework and activities without fees. Access to coursework was provided through Niswonger Online where students in middle school could take Algebra 1. Research Design: The impact evaluation used a school-level randomized controlled trial design. Data Collection and Analysis: Administrative data were obtained from the Tennessee Education Research Alliance and included measures of student test scores, courses taken, course grades, attendance, demographic information, school culture and climate, school demographics, and schoolwide achievement. Data were analyzed using multilevel models with students nested within schools. Findings/Results: Overall, after two years of implementation, students in schools that were receiving the STEM.LD intervention did not perform significantly differently than students in the comparison group schools. The impact was positive but not statistically significant for math (0.61, p=0.928) and positive but not statistically significant for science (4.54, p=0.324). These results could be attributed to several reasons. First, after difficulty in recruiting schools to participate, the research design shifted to a multiple cohort model. However, recruitment efforts for a second cohort were not as fruitful as hoped, and the study was underpowered. Part of the recruitment challenge was that there were many competing programs and projects in the region. Second, although STEM.LD was designed to give schools flexibility in implementation to address their unique needs, this flexible approach resulted in wide variation in experiences and implementation across schools. Rather than specifying minimum amounts or thresholds that schools needed to participate in, schools were given autonomy in how they used STEM.LD resources. Third, the development and implementation of a platform listing available STEM programs, activities, and opportunities was delayed. Conclusions: The STEM.LD program was not effective in increasing middle school student mathematics and science achievement. But there were many lessons learned to apply to other innovative projects moving forward that might help improve success. The most important lesson was ensuring sufficient time for planning and specifying the options or components that schools would need to participate. When schools and leaders are bought into the intervention, they are more likely to plan for integrating the intervention with their other initiatives.
Descriptors: STEM Education, Student Centered Learning, Faculty Development, Middle School Students, Science Achievement, Mathematics Achievement, Grade 7, Grade 8, Standardized Tests, Public Schools, Program Effectiveness, Educational Research, Educational Innovation, Grants, Program Evaluation, Mathematics Instruction, Science Instruction, Intervention, Grade 6, Science Teachers, Mathematics Teachers, Learner Engagement, Instructional Leadership, Partnerships in Education
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Related Records: ED677188
Publication Type: Reports - Research
Education Level: Junior High Schools; Middle Schools; Secondary Education; Elementary Education; Grade 7; Grade 8; Grade 6; Intermediate Grades
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Tennessee
Grant or Contract Numbers: N/A
Author Affiliations: N/A

Peer reviewed
Direct link
