Chapter 1: Studies Show Mixed Results, Spur Calls for Changes in Program
Although Chapter 1 has funneled more than $60 billion in federal funds to school districts over the past 25 years, research does not unequivocally back up lawmakers' apparent faith in the compensatory-education program.
"After more than two decades of study, evaluators have declared it to be 'a program that works,' a program which has produced modest gains, and a virtual waste of taxpayers' money," a group of researchers recently wrote in a paper summarizing the research literature on Chapter 1.
The paper, obtained in draft form by Education Week, was written for the Education Department in preparation for a massive, unprecedented longitudinal study of Chapter 1 students that aims to answer some of the outstanding questions about the program's efficacy. The authors collaborated on the study design and hope to win a contract to actually conduct it as well.
"The research that exists to date basically says Chapter 1 does work in the sense that it helps kids do better than they otherwise would have done," said Robert E. Slavin, a researcher at Johns Hopkins University who is widely considered a national expert on Chapter 1.
But, he added, the research also shows that the program "doesn't work well enough to help them catch up with their more advantaged peers.''
"Has it been a waste of money? Not at all," said Mr. Slavin, director of the elementary-school program of the Center for Research on Effective Schooling for Disadvantaged Students. "I think most people would say it's made a difference, but when you look at the picture of what poor kids, minority kids are doing, you have to say we need to do something different."
Stanley Pogrow, an associate professor of education at the University of Arizona who has designed an acclaimed, computer-based program to teach thinking skills to Chapter 1 students, offers a blunter assessment.
"Let's face it: Chapter 1 is ineffective," he said. "The kids you're serving never get out of Chapter 1. The problem isn't with the concept, but with the way school districts use the money."
Studies Show Mixed Results
Data from the National Assessment of Educational Progress show that the achievement of disadvantaged and minority children has improved relative to that of the general population since the inception of Title I in 1965. According to the "National Assessment of Chapter 1," which was mandated by the Congress in preparation for reauthorization of the program in 1988, that trend became particularly evident with children born after 1963, who would have entered school in the late 1960's, when the federal program was becoming fully operational.
However, studies specifically designed to measure the impact of Chapter 1 programs show mixed results, with most finding modest gains.
"Even when you look at what Chapter 1 calls its exemplary programs, you don't find that they systematically bring children into the mainstream, even over several years," said Henry Levin, a professor of education and economics at Stanford University and the director of its Center for Educational Research. "You find that a kid who's in the 15th percentile comes up to the 20th percentile."
The only comprehensive national study to date on the achievement of Chapter 1 students is the "sustaining-effects study," which was commissioned by the federal government in the late 1970's.
Researchers collected achievement data on 120,000 students in 243 schools in 1976-77, then followed a smaller group of students over two subsequent years. It found that students served by what was then called Title I generally improved more over the course of a year than did other "needy" students who did not receive services.
Children in mathematics programs and students in the early grades showed the greatest gains, while reading students in grades 4 through 7 did not improve appreciably faster than the comparison group.
Moreover, the limited longitudinal data collected for the sustaining-effects study indicated that participating children eventually lost much of the ground they had gained after exiting from the Title I program.
The most recent "National Assessment of Chapter 1," which was completed in 1987, included re-analysis of the "sustaining effects" data that used new statistical techniques to form comparison groups more similar to the Title I students in the sample. The researchers found that "the more similar the comparison group was to Title I participants, the greater the achievement benefits associated with Title I participation."
Test Scores Analyzed
The national assessment also analyzed test scores collected by school districts, which showed that the average Chapter 1 student improved his national percentile ranking slightly over the course of the 1983-84 school year.
The authors concluded that Chapter 1 students achieved greater increases in their test scores than comparable students not participating in the program, but that "their gains do not move them substantially toward the achievement levels of more advantaged students."
Researchers also say that existing studies offer no conclusive evidence for the superiority of either "pullout" instruction, for which Chapter 1 students leave their regular classrooms, or in-class assistance.
"Based on the research that has been conducted to date, we cannot conclude with confidence either that pullout is more effective than in-class instruction or that the opposite is true," said the recent research summary, which was prepared by the group of contractors who collaborated on designing the upcoming longitudinal study.
While researchers acknowledge that studies to date have failed to find large gains by Chapter 1 students, they hasten to point out that the lack of sharp results could be due, at least in part, to the difficulty of measuring the program's impact.
The researchers cited several difficulties, including:
- The variability of Chapter 1 programs.
Federal regulations set some guidelines, but schools have a great deal of leeway to design their own instructional programs for Chapter 1 students. Students in different schools can receive very different services, and those services can be offered at a wide range of intensity levels.
- The exceptional mobility of the Chapter 1 population.
Many children cannot be retested over time, and it is impossible to determine which school's program is responsible for the gains some mobile students make. The children who are most successful leave the program and often are not retested, possibly causing an understatement of the program's impact.
- Isolating program effects from the impact of a child's regular school program.
- Lack of adequate control groups.
Probably the thorniest problem facing researchers trying to evaluate Chapter 1 is the difficulty of finding a similar group of students not receiving services with whom Chapter 1 students can be compared.
"It's hard research to do because most of the kids who qualify for Chapter 1 services are receiving them," Mr. Slavin said.
By definition, students receiving Chapter 1 services are needier than classmates who do not, and schools receiving Chapter 1 funds serve a more disadvantaged population than others in the same district that do not receive funds.
Comparing students in different schools and schools in different districts means having to ensure that results are not skewed by differences in demographics, resources, and school philosophy unrelated to Chapter 1.
"It's costly to do; it's hard to do; it's a complex issue, and it takes considerable effort and thought," said Elois Scott, who is overseeing the longitudinal study for the Education Department's planning and evaluation service. "But our data show that there are similar kids in similar schools who may or may not be receiving services."
According to study documents, Ms. Scott, and Mr. Slavin, who contributed to the study design, it aims to circumvent these problems by using several different approaches at once, comparing:
- Schools and students that are near the "cutoff point" for Chapter 1.
In a given school, only a certain number of children can be served, and those whose achievement is either barely high enough to place them out of Chapter 1 or barely low enough to qualify them can be compared. Likewise, some districts have two similar schools, one with just enough low-income students to get Chapter 1 funds and one with barely too few.
The disadvantage of this approach is that it does not study the most disadvantaged children.
- Comparing similar children who received reading instruction with those who received help in math.
- Trying to match students with similar characteristics in different schools while controlling for differences between schools.
Ms. Scott said 247 participating districts and about 300 schools have been chosen in 46 states. She said districts were chosen to give the most representative sample, and that those reluctant to participate will essentially be forced to do so.
"We're trying to encourage them, because the study is so important, not only for Chapter 1 but for what it can tell us generally about the education of disadvantaged children," Ms. Scott said.
However, she said, "it is our interpretation" that Chapter 1 regulations require districts to provide whatever information is deemed necessary.
Data collection is to begin this spring. The study team will administer standardized tests to children in grades 1, 3, and 7. The youngsters will also be surveyed about their school and extracurricular activities.
Several Surveys Planned
Teachers, administrators, and parents will be surveyed about "kinds of services, level of service, coordination between programs, the type of curriculum used in the regular program and how it relates to Chapter 1, emphasis on higher-order skills, attitudes," Ms. Scott said.
In addition to academic achievement, the study will track trends in delinquency, retention, school grades, and dropout rates.
The researchers plan to follow the children over several years as well as to create a larger longitudinal sample by matching similar children in the different age groups.
Ms. Scott said they will try to follow all children who move within a district and all dropouts, and will follow as many of those who move to another district as possible.
"Existing research has not followed students or looked at dropouts,'' she said. "It also hasn't taken a single group of kids and followed them all the way through."
The data will be sufficient to make comparisons between major ethnic groups, and an additional grant from the department's office of bilingual education and minority-languages affairs will enable researchers to sample enough limited-English-proficient students to draw conclusions about them. The department decided, however, that adequately sampling Indian students would be too expensive, Ms. Scott said.
The Congress has authorized a total of $22 million for the study.
"A lot has happened since the sustaining-effects study--the whole reform movement and emphasis on higher-order thinking skills," Ms. Scott said. "I think we may find something very different about academic achievement and how it is sustained in these kids."
In addition to the longitudinal study, which is not due until 1997, the Education Department is beginning a new "national assessment" to be completed before the Congress is to reconsider Chapter 1 in 1993.
The law authorizing the study specifically mandated an independent advisory panel and prohibited the department from altering the contractors' work.
That study is to examine:
- The implementation and efficacy of new provisions written into the 1988 reauthorization law, including rules for operating school wide projects, parent-involvement requirements, and program-improvement provisions that require remedial action to improve programs whose students show insufficient academic gains.
- How Chapter 1 funds are allocated, how children are chosen, and the number of eligible children not being served.
- The qualifications of Chapter 1 instructors.
- The effectiveness of Even Start, which combines adult literacy and parenting programs with preschool for young children, and of programs for migrant students.
- Student achievement, "as reflected by student attendance, behavior, grades, and other indicators of achievement."
Vol. 10, Issue 35, Page C8-9Published in Print: May 22, 1991, as Chapter 1: Studies Show Mixed Results, Spur Calls forChanges in Program