A new study on using data to improve after-school programs finds that people and processes are just as important as technology.
Collecting data has become increasingly important for these programs both to show progress and, in many cases, to help justify their funding.
“It is a way for after-school providers to understand who they’re reaching, the quality of their programs, the accessibility of the programs,” said Julie Spielberger, the study’s lead author and a research fellow at Chapin Hall with the University of Chicago.
The Wallace Foundation commissioned the study, called, “Connecting the Dots: Data Use in Afterschool Systems,” which examined such systems in the following cities that are part of the nonprofit’s Next Generation After-School System-Building initiative:
- Baltimore, Md.
- Denver, Colo.
- Fort Worth, Ind.
- Grand Rapids, Mich.
- Jacksonville, Fla.
- Louisville, Ky.
- Nashville, Tenn.
- Philadelphia, Pa.
- Saint Paul, Minn.
This initiative is designed to improve the systems that help children from low-income families access and participate in high-quality, after-school programs.
Researchers from Chapin Hall sought to find out how these cities, “build their capacity to capture, describe, and improve their practices through their data systems.” They examined the cities between 2012 and 2014 by conducting site visits and interviews with key stakeholders both in person and by phone. They also sat in on selected training sessions and meetings related to data use, studied city documents and attended three cross-city grantee meetings. They released their preliminary findings in late May and continue to study the issue.
During the two-year initial study period, all nine of the cities were in various stages of collecting data and using it to inform their after-school infrastructure, improve programs and decide how best to allocate resources.
In the past, data collection for after-school programs primarily focused on compliance. For example, a program would need to show a funder that it served a certain number of students for a certain number of hours. But, today, a program would use that data to look deeper into their offerings.
“The program would want to take that same data and then look at it and say on Mondays the majority of my kids aren’t coming, but they come Tuesdays, Wednesdays and Thursdays,” said Jennifer Axelrod, one of the study’s authors and a policy fellow at Chapin Hall. “What does that tell me about my program? I want to look at that data in real time or on a weekly basis, so that I can continue to make refinements if kids are coming late, if they’re leaving early. All of that same data can be used for compliance purposes, but if I’m a program, I want to use it to understand, am I meeting my mission as an organization to serve the children as well as I can and to understand and address barriers that might be affecting participation.”
Three Main Components
The researchers used a triangle to represent data systems. The three sides were represented as people, processes, and technology. Technology includes the computers and software used to collect and process data through management information systems. But how that is done comes down to the people involved.
“People are doing much of that work to get the data into the MIS and taking the data to ensure its quality and then analyzing it and interpreting it,” said Spielberger.
People also drive the processes behind data systems. They decide how workers will be trained to collect the data, how reports will be produced and how decisions will be made based on their findings.
Whenever organizations collect student data they have to be mindful of laws regarding student privacy.
“School districts are rightfully so very concerned about protecting the educational records of their students and being good stewards of that data,” said Axelrod. “But it can cause some very significant barriers as we want to talk about the impact and improvement. Where we’ve seen this work really well is helping people understand where they can use aggregate or combined level data. That means that you don’t have individualized levels of information about young people, so you can’t identify any one student. But you can understand what’s happening with your program at a subgroup level. So 10 or more students—a lot of times that’s enough data, and that level of comfort gives schools a little more willingness to share data.
The researchers identified some best practices when it comes to building data systems: starting small, providing ongoing training, and accessing help from experts.
A number of cities in the study began with modest goals for data collection and use and then planned to expand slowly. Since after-school programs tend to have a lot of staff turnover, having plans in place for ongoing training also proved to be essential.
“Organizations that are supporting this work are often providing the one-on-one trainings on a regular basis and then also trying to provide deeper level training for those that are staying with the process, so that they can really think through going from data collection to data for program evolution,” said Axlerod. “The ability to apply and use data takes a significant amount of coaching.”
Since this is a specialized skill, many cities turned to outside help.
“Sometimes, the expertise came from partner organizations, other research entities—whether they were universities or independent data organizations in the community,” said Spielberger. “Other cities without these resources would hire someone to work within the intermediary or network organization of the after-school system.”
The researchers are continuing to follow these cities and plan to release a second report when the study ends in 2017. In the meantime, they’ll be examining how other cities are handling data systems related to after-school programs.
“We’re really interested in what kinds of uses of data develop over the next couple of years,” said Spielberger.
They also plan to look into the true costs of these systems going beyond the prices of computers and software into the time it takes to implement these systems and the work hours required.
Axlerod calls this an exciting time for researchers interested in the impact of data systems, particularly as the Every Student Succeeds Act pushes for increased family engagement.
“We’re uniquely poised to really use data and the system building that is evolving through collective impact to create transformations and engagement of the families and youth that we most want to serve,” said Axlerod.
A version of this news article first appeared in the Time and Learning blog.