Our Projects
Intervention Program Evaluation in a Midwest Urban School System
The Challenge
In 2022–24, an urban school system in the Midwest asked whether the students its new reading and math interventionists served did better than similar students who were not served, and whether interventionists were teaching the way the district expected.
The district hired the interventionists to work with small groups of students whom a universal screener placed in the highest tier of need. Each interventionist served several groups four to five days a week. School staff chose which of those students to serve, using different data and decision rules from school to school.
Our Support
Gibson Consulting Group estimated the program's effect with a quasi-experimental design, observed intervention lessons, and surveyed interventionists, delivering two reports on research questions set with district staff.
Impact analysis. Gibson weighted all students not served to resemble intervention students on grade, demographics, program status, and beginning-of-year scores, then compared end-of-year results. Because staff chose students partly on information the weights could not include, the comparison was approximate.
Classroom observations. In Fall 2023, the team completed 53 observations of instruction carried out by 40 interventionists using a rubric chosen with district staff.
Survey and interviews. Gibson surveyed interventionists, receiving 44 responses, and interviewed district staff.
Gibson's project lead was Dr. David Osman.
Results
The study found no evidence that the intervention raised 2022–23 end-of-year reading or math scores relative to the weighted comparison group. At the end of 2022–23, intervention students' reading scores were two points above the comparison group's, a difference that was not statistically significant. In math they were two points below, a statistically significant difference too small to matter in practice.
Of the 1,300 reading intervention students analyzed, 14% met the year-end reading benchmark, compared with 24% of the weighted comparison group; in math the shares were 10% and 22%.
On the state assessment, about 5% of intervention students in Grades 3 through 8 met the benchmark, compared with about 11% of the comparison group.
In 34 to 44 of the 53 observations, depending on the practice, observers rated the district's expected practices as fully demonstrated.
The report recommended developing common evidence-based materials, a fidelity monitoring system, a consistent rule for choosing which students to serve, and a common student data collection tool.
For your district. A district paying for intervention staff can use an evaluation of this kind to decide which students to serve, which materials to standardize, and how to monitor delivery.