Holding after-school programs accountable can be a challenge, according to some after-school leaders. Difficulties retaining staff and a lack of concrete assessment objectives have left many programs without the tools to evaluate themselves.
Programs receiving grants from California’s After School and Education Safety Program (ASES) are held to some accountability measures that can factor into whether their grants are renewed. The ASES program, which spends $550 million annually supporting after school at 4,000 schools in the state, requires sites to submit the attendance records of participants and some method to evaluate student outcomes, either standardized-test scores or a measure of student-behavior change, homework completion, or skill development.
Because standardized scores are more readily accessible, in the past, most sites have submitted test scores as a measure of student outcomes, said Deborah Lowe Vandell, chair of the education department at University of California, Irvine. But the state department of education is currently supporting efforts to provide sites with other measures to assess the impact ASES programs have had on youths.
Lowe Vandell is now working on a project funded by the state department of education and the David & Lucille Packard Foundation called the California Afterschool Outcome Measures Project (CAOMP). It will assess, through an online data system called the “online toolbox,” whether after-school programs have affected student performance and behavior.
Sites can opt to submit student self-surveys or staff or classroom-teacher reports on student performance (or all three) for assessment. The survey questions, which ask, for example, about student persistence, independence, and socialization, were created after reviewing literature on what student behaviors were indicators of improvement that could be influenced by a quality after-school program.
“There are other ways to look at student outcomes other than standardized-test scores,” Lowe Vandell said. “As we start to measure additional behaviors consistently and reliably and they are validated against other outcomes we care about, they can help develop a broader array of measurement tools that drive [after-school] programming.”
Last fall, the outcome-measures project collected data through the online toolbox from 157 sites, totaling 6,000 students, 4,300 staff members, and 670 teachers. Spring data will be pulled next month, and by summer, programs will be provided overall report assessments on positive/negative changes in their student outcomes over the school year. Sites can see how their assessments stack up against their site peers throughout the state, and use the data to fulfill the accountability requirements requested by the state department. Project leaders hope to open the toolbox to after-school programs nationwide by the fall, though will eventually charge a modest fee for usage.
The California AfterSchool Network, an organization that works to support state after-school programs and favorable out-of-school-time policies, sees a link between student outcomes and the general quality of after-school programs.
Since the rapid expansion of after-school programs in California as a result of the grant program, many new sites have struggled with having the know-how to assess the quality of their programs and improve them, said Jeff Davis, a program coordinator with the California AfterSchool Network. The data currently required for the state department of education examines more general student-performance markers and less program quality components that can lead to improved student outcomes, he added.
With collaboration among state after-school leaders, the California AfterSchool Network has developed an online “Program Quality Self-Assessment Tool” that will help with a staff-directed evaluation process. The tool helps programs assess themselves on 11 criteria they deemed important to deliver a well-rounded program by measuring how well programs are meeting those objectives in tiered levels. Categories include community partnerships, school alignment, staff professional development, diversity and inclusion, and English-learner support.
Most of the criteria evaluators are applicable to after-school programs throughout the country, Davis said, other than the English-learners objective, which is more geographically specific.
Read more about after-school programs in California in my article this week on LA’s BEST, a Los Angeles-based after-school organization that receives significant support from the ASES program.