School & District Management

After-School Report Called Into Question

By Linda Jacobson — May 21, 2003 4 min read
  • Save to favorites
  • Print

A study that influenced President Bush to recommend a big cut in spending for a federally financed after-school program has serious flaws, according to a statement by seven members of the study’s technical working group.

The first-year report on the 21st Century Community Learning Center program, conducted by Mathematica Policy Research, has “methodological problems that call into question its findings and that violate basic principles governing how evaluation should be used to guide policy and affect program budgets,” the statement contends.

Mathematica Policy Research posts a response to the statement issued by some members of the study’s technical working group. (Requires Adobe’s Acrobat Reader.)

Officials of the U.S. Department of Education, which paid for the Mathematica study and has championed the value of scientifically based research in determining the effectiveness of education programs, said they “stand behind the study.”

The principal investigators for the study, Mark Dynarski and Mary Moore, said in their own response last week that the technical advisers had had sufficient time before the study was released to raise their concerns, and that the study had never suggested that funding for the $1 billion program be cut.

President Bush is requesting a 40 percent reduction in the program for fiscal 2004.

The study’s only policy recommendations, Mr. Dynarski and Ms. Moore said, were that programs look for ways to increase student participation and improve academic content.

Mr. Dynarski added in an interview that one of the members who signed the statement—which was e-mailed to the media last week—had hardly participated in the meetings, and that two other members did not sign the statement.

“It feels political to me,” he said. “I think they got caught up in the moment of the budget cuts.”

Christopher T. Cross, a member of the nine-person working group and a senior fellow at the Washington-based Center on Education Policy, responded that the group never saw the final version. Mr. Cross, one of the signers of the statement, served as an assistant U.S. secretary of education for research during the first Bush administration.

The debate over the Mathematica findings indicates that more work needs to be done to determine the best way to design and evaluate after-school programs, said Megan Beckett, a social scientist working in this field at the RAND Corp., a think tank based in Santa Monica, Calif.

“It’s a rough field, and it’s just now starting to be held accountable,” she said. She added that the Mathematica study is “the most rigorous so far.”

Deputy Secretary of Education William D. Hansen, meanwhile, testified before a Senate Appropriations subcommittee last week to further explain the administration’s proposal to cut funding for the initiative by $400 million. One reason, he said, is that the program grew “rapidly with little consideration of its effectiveness.”

‘Key Measures’ Questioned

Released in February, the study by Princeton, N.J.- based Mathematica concluded that the program had had few positive effects on student achievement and had not led to improvements in students’ behavior or feelings of safety. (“Study Critiques Federal After-School Program,” Feb. 12, 2003.)

The seven working-group members who released the statement say some of those conclusions were not justified, however.

For example, they say, the study did not reflect the fact that middle school students in the control group started out scoring higher than those who were being served by the program.

“In our judgment, the first-year analyses and any subsequent analyses are un-interpretable given these substantial baseline differences on key measures,” the statement says.

But the Mathematica researchers countered that while the treatment and control groups were different in some respects, the study used “statistical methods specifically designed to adjust for the differences.”

In the elementary school evaluation, the seven working-group members say, the size of the study sample was so small that they do not have much confidence in the results.

Data were collected on 403 students in the treatment group and 226 students in the control group from 18 elementary sites. But in the 2001-02 school year, there were more than 6,800 21st Century Community Learning Center sites operated by more than 1,400 grantees, offering a variety of activities such as homework assistance, recreation, and intellectual enrichment.

The statement notes that the researchers later added 1,600 more students to the sample, but still reported the findings on the smaller sample.

Mr. Dynarksi and Ms. Moore argue, however, that while a larger sample would have been preferable, their study is still the first one on after-school programs to use random assignment. And they noted in the report that the findings were preliminary.

Analyzing Outcomes

Whether academic improvements should have been expected from all of the sites is also a point of contention between the seven working-group members and the researchers. Four of the elementary sites in the study offered programs aimed at adults, such as efforts to improve job skills, and the children attended only with their parents or grandparents.

“It is not clear why or how these adult-focused programs would be expected to impact child outcomes,” the group’s statement says.

But the researchers argue that the approach used at those sites was “based on the hypothesis that by supporting parents, programs can achieve better academic outcomes for students,” and understanding whether that was accomplished was an important objective.

The working-group members also say they found inconsistencies between the full report and the executive summary, a contention that the researchers dispute. And the members say in their statement that positive outcomes might not have been found because students didn’t participate regularly enough—a factor that other after-school studies have concluded is necessary for results to be achieved.

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
IT Infrastructure Webinar
A New Era In Connected Learning: Security, Accessibility and Affordability for a Future-Ready Classroom
Learn about Windows 11 SE and Surface Laptop SE. Enable students to unlock learning and develop new skills.
Content provided by Microsoft Surface
Classroom Technology K-12 Essentials Forum Making Technology Work Better in Schools
Join experts for a look at the steps schools are taking (or should take) to improve the use of technology in schools.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Budget & Finance Webinar
The ABCs of ESSER: How to Make the Most of Relief Funds Before They Expire
Join a diverse group of K-12 experts to learn how to leverage federal funds before they expire and improve student learning environments.
Content provided by Johnson Controls

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Opinion 7 Ways Principals Can Support Teachers
Listening more than talking is one vital piece of advice for school leaders to help teachers.
13 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
iStock/Getty
School & District Management What Schools Can Do to Tackle Climate Change (Hint: More Than You Think)
For starters, don't assume change is too difficult.
7 min read
Haley Williams, left, and Amiya Cox hold a sign together and chant while participating in a "Global Climate Strike" at the Experiential School of Greensboro in Greensboro, N.C., on Friday, Sept. 20, 2019. Across the globe hundreds of thousands of young people took the streets Friday to demand that leaders tackle climate change in the run-up to a U.N. summit.
Haley Williams, left, and Amiya Cox participate in a Global Climate Strike at the Experiential School of Greensboro in Greensboro, N.C., in September 2019.
Khadejeh Nikouyeh/News & Record via AP
School & District Management 'It Has to Be a Priority': Why Schools Can't Ignore the Climate Crisis
Schools have a part to play in combating climate change, but they don't always know how.
16 min read
Composite image of school building and climate change protestors.
Illustration by F. Sheehan/Education Week (Images: iStock/Getty and E+)
School & District Management Some Districts Return to Mask Mandates as COVID Cases Spike
Mask requirements remain the exception nationally and still sensitive in places that have reimposed them.
4 min read
Students are reminded to wear a mask amidst other chalk drawings on the sidewalk as they arrive for the first day of school at Union High School in Tulsa, Okla., Monday, Aug. 24, 2020.
Chalk drawings from last August remind students to wear masks as they arrive at school.
Mike Simons/Tulsa World via AP