Summer programs in large urban school districts showed benefits for mathematics but flat results in reading and social emotional development, according to preliminary findings released today, from the first longitudinal study of such initiatives.
The RAND Corporation study, known as the National Summer Learning Project, provides data from the first summer of a six-year, randomized controlled trial of programs in Boston; Dallas; Duval County, Florida; Pittsburgh and Rochester, New York. The study was funded by The Wallace Foundation, which also provides funding to Education Week for coverage of extended learning, leadership, and arts education.
Its report, “Ready for Fall?” finds that low-income students who voluntarily attended a district-led program in the summer between third and fourth grade, showed gains in math skills equal to about 20 percent of a typical student’s growth during one year.
Prior research on summer programs was fragmentary and limited by size and length, said Edward Pauly, director of research and evaluation at the Wallace Foundation. There wasn’t enough research-based evidence to inform school districts or policy makers when deciding how to prioritize funding.
“There’s a big question that we’re now beginning to find the answer to about whether large, urban districts can successfully operate these programs on a large scale in a way that makes a difference for kids, and the math finding shows that that’s in fact the case,” Pauly said. “Combined with the fact that this happened in not one, but five districts, this is a very robust, powerful finding that tells us something that is likely to be very generalizable.”
Research has shown that low-income students lose more academic ground during the summer than their wealthier classmates. As Education Week reported in October 2014, the gap between the most academically at-risk students and better-off peers widens by about 2.6 months in reading and 2.7 months in math during the summer months.
This study compared 5,637 students who completed 3rd grade in the spring of 2013 and applied for their district’s summer program. Of those, 3,194 were randomly assigned to the group that attended the summer program, while another 2,445 students were assigned to a control group and not offered a spot in the program. Nearly half in each group were African-American students, 40 percent were Hispanic students, and 89 percent were eligible for free- or reduced-price lunch (a common proxy for measuring poverty).
Students in both groups were tested in math and English/language arts when they returned to school in the fall of 2013. Their teachers also completed a survey asking about each student’s social-emotional skills.
“Initially, I was surprised at the increase in math, because it was more significant than I expected, and disappointed at the results in reading,” said Christine Cray, project manager of the Summer Dreamers Academy in Pittsburgh Public Schools.
But Cray added that having access to this data is a key reason she wanted the Pittsburgh district to participate in the study.
“The opportunity to get formative feedback from Rand about our program quality and implementation is invaluable,” Cray said.
RAND researchers observed the summer programs at least once and evaluated them on a variety of measures of instructional quality, including the amount of time spent on academics, how well teachers knew the content, student behavior and teachers’ enthusiasm.
Study participants’ improvements in reading depended on several key factors, said Jennifer Sloan McCombs of RAND, who co-authored the report. Students in both the treatment and control groups did better than their peers when they had teachers with experience in reading instruction in 3rd or 4th grades and those who were more adept at dealing with disruptive behavior in class.
Wallace selected the five districts from a pool of 35 that were already running summer programs. In order to participate, each district had to agree to common elements—subsidized by Wallace—to ensure that the research comparisons would be valid. Each summer program must:
- Run voluntary, full-day sessions with a total of at least three hours a day of instruction in math and reading taught by certified teachers;
- Include enrichment activities;
- Have no more than 15 students per class;
- Have no fees to participate;
- Provide free transportation;
- Provide free breakfast, lunch, and snacks; and
- Last a minimum of five weeks.
Initially, the study was going to run for four years at a cost to Wallace of $50 million, said Pauly. But it was expanded to six years after districts reported wanting it to continue. Wallace is expected to contribute an additional $5 million. That means the support will be reduced, although it was never the foundation’s intention to provide ongoing funding.
All the districts had found ways to support their summer programs before the study, but funding remains a challenge. In Pittsburgh, the Summer Dreamers Academy had stimulus funding from the American Recovery and Reinvestment Act that allowed them to accept 5,400 students in the summer of 2010. When that money ran out, enrollment dropped to under 2,000 last summer. The district has prioritized students into three groups based on those most at risk of summer-learning loss, explained Cray, “but every year, regrettably, we do have to send out a significant stack of, ‘Sorry, you’re not able to attend our program because we’re already full’ letters.”
When students in the control group were asked what they did over the summer, McCombs said 60 percent reported that they didn’t participate in any organized activity, even though their families were given a list of city and community groups running summer programs.
Dallas Independent School District has decided to expand its investment in all its summer programs, including Thriving Minds Summer Camp, which is part of the Wallace study, Superintendent Mike Miles said. They’ve known for a while that kids who come from disadvantaged environments often need support in the summer to extend their learning and do enrichment, said Miles, adding that this study is “yet another piece of evidence that we can, as a school system and as a community, not have regression and actually advance student learning over the summer if we make the right investments.”
Miles said the district is also increasing the rigor of all its summer programs.
“It’s not just time on task: It’s the quality of instruction; it’s the expectations; it’s the rigor; it’s high-quality teachers delivering high-quality instruction,” he explained.
RAND is already working on the second report for the National Summer Learning Project, that will include results of the students’ scores from the spring 2014 assessment along with their grades and attendance records to see how long the summer effects lasted. Those results are scheduled to be released in spring 2015.
Additionally, students in the treatment group were invited back to attend their district’s program this past summer, right after they completed fourth grade. Rand will also be reporting on their outcomes after two years of participation.
Although these are early results, McCombs said they have implications for how districts might improve their programs. “This is one sliver of information of a much larger chapter,” she acknowledged. “If we’re reading a book we’re in chapter one; and we’re going to eventually get to chapter five or six, and at that point I think it will be a lot more revealing as to whether things are really exciting or unexpected.”
A version of this news article first appeared in the Time and Learning blog.