A new study of KIPP, a large charter network serving mainly low-income black and Hispanic students, finds that its schools continue to have a positive impact overall on student achievement, and yet they show no effect on student motivation, engagement, or behavior.
Elementary and middle school students at KIPP, or the Knowledge Is Power Program, schools had significantly larger gains on reading and math tests than their peers at non-KIPP schools, according to the study conducted by Mathematica Policy Research.
At the high school level, the impact on student achievement was only statistically significantly positive for those students who were new to the KIPP system. Going to a KIPP high school did not have an added benefit for students who had attended a KIPP middle school.
The KIPP network has expanded rapidly in recent years, going from 45 schools in 2005 to more than 180 nationwide today. In 2010, the KIPP Foundation received a $50 million federal Investing in Innovation grant, which allowed it to double the number of students it served over a five-year period. The study released today, which is the final report in the long-running Mathematica evaluation, aimed to see how the expansion affected school quality.
“As KIPP has scaled, the network has continued to demonstrate the kinds of positive impacts demonstrated in previous studies,” Christina Tuttle, the lead author for the Mathematica report, said in a Sept. 16 webinar for journalists.
A 2013 Mathematica studied showed that KIPP middle school students experienced significantly greater learning gains in math, reading, science, and social studies than their peers in traditional public schools.
For the new study, the researchers gathered data from 8 elementary, 43 middle, and 18 high schools using a combination of lottery-based and quasi-experimental designs. They looked at results from state-administered assessments, study-administered assessments, and student and parent surveys.
At the elementary level, the impact of getting in to a KIPP school was, after two years, equivalent to improving a student’s score on a reading test from the 78th to the 84th percentile. In math, KIPP elementary students scored at the 68th percentile on a calculation test compared to their non-KIPP peers, who scored at the 58th percentile.
(It’s worth noting that students who were chosen by lottery to attend KIPP schools but did not enroll were considered KIPP students for this study. Those who entered the lottery but were not chosen were part of the control. The aim was to ensure “that treatment and control group students are similar at baseline (pre-KIPP) in terms of demographics and academic preparation as well as key factors such as motivation and parental support,” the researchers write.)
Impact for Middle School Students Declines
For middle school, the researchers were able to look at student achievement over a 10-year period. They found that overall KIPP middle school students improved more in math, reading, science, and social studies than their peers at non-KIPP schools.
The impact of getting into a KIPP middle school was equivalent to a student moving from the 37th to the 44th percentile in reading over two years. In math, it was equivalent to moving from the 40th to the 50th percentile.
But the size of the impacts in math and reading declined from 2005 to 2014. (The green line, below, indicates the number of KIPP charter schools.)
“Undoubtedly, the largest impacts occurred in the earliest years of KIPP,” said Philip Gleason, the study’s principal investigator, during the webinar. The effect size peaked in 2006 and declined, yet remained statistically significantly positive, from there.
During the five years of the federal i3 grant (2010-2014), the number of schools expanded rapidly but the effect size “remained fairly steady,” noted Gleason.
Looking at the middle school results based on when the schools opened shows that the impact on students dipped between 2006 and 2010, and then rebounded.
Steve Mancini, KIPP’s director of public affairs, noted by email that in 2007, coinciding with the dip in effect size, the network began opening more elementary and high schools. “It is more challenging opening three different types of schools—middle, elementary, and high—than just one. We have reached an equilibrium, but it took us time to find our footing.”
No Effects on Motivation
Perhaps the most surprising finding from the report was this: KIPP schools had no statistically significant impact on most measures of student motivation, engagement, behavior, or educational aspirations.
KIPP schools are well-known for their efforts in character education (their motto is “Work Hard. Be Nice”) and they place a lot of emphasis on college preparation.
The data on these student behaviors and attitudes were gathered through parent and student surveys. Regarding motivation and engagement, the researchers asked about students’ academic confidence, grit, self-control, how much time they spent on homework, and how much effort they put into school. For behavior, they asked about peer pressure, illegal activities, and both positive and undesirable behaviors.
The researchers said one explanation for these findings could be reference bias. “One of the limitations of trying to measure impacts based on student surveys is that students oftentimes report on their behavior using a reference group,” said Gleason. “In general, KIPP students compare themselves to other KIPP students.”
Mancini furthered that thought. “The standard at KIPP for hard work is candidly a lot higher than at a typical school,” he said. “It may be that the bar in the comparison group at schools is just not as high.”
KIPP students weren’t more likely to have high aspirations for college attendance and completion than their peers. But they were more likely to participate in college-preparation activities, such as having discussions about college, getting assistance in planning college, and applying to at least one school.
A version of this news article first appeared in the Charters & Choice blog.