Two studies of the first program in the nation that allows children from poor families to attend private schools on public dollars have reached dramatically different conclusions about whether it raises student achievement.
Both sides in the bitter and politically charged debate over school choice have seized on the conflicting analyses of the 6-year-old Milwaukee voucher program. And the senior authors of the two studies have publicly criticized each other with unusual rancor, in part because of the high stakes.
The controversy pits a University of Wisconsin political scientist who has studied the Milwaukee program since its inception against researchers from Harvard University and the University of Houston who published their analysis in August.
The newer study concludes that students who were enrolled in the program for at least three years outperformed a comparable group of children who remained in the Milwaukee public schools. The earlier research found “mixed results” and no significant gains in test scores. It has been widely cited by voucher opponents as evidence that the program doesn’t work.
The bottom line, education experts say, is that neither study contains the conclusive, long-term evidence needed for making broad policy decisions. “It’s going to take a lot more analysis than this to figure out what’s going on,” says Richard Elmore, a professor at the Harvard graduate school of education.
The new research comes amid heightened interest in school vouchers nationwide. Republican presidential nominee Bob Dole has said a federal voucher program would be his top education priority if elected. And governors and legislators in several states are considering similar plans. They have their eyes on the Milwaukee experiment and a similar effort that got underway this fall in Cleveland.
Unlike in Milwaukee, the Cleveland program allows parents to redeem vouchers at private religious schools, despite opponents’ attempts to persuade the courts to stop it. (“A New Choice,” this month.) A 1995 Wisconsin law to open Milwaukee’s program to religious schools is on hold while the courts decide if it is constitutional.
“There are enormous amounts at stake,” says John Witte, the University of Wisconsin-Madison researcher who led the initial study of the Milwaukee program.
Witte’s analysis, conducted at the request of the Wisconsin education department, has come under harsh attack. In an August 14 article in The Wall Street Journal, Paul Peterson, director of Harvard’s program in education policy and governance, and Jay Greene, an assistant professor of political science at the University of Houston, wrote that Witte’s study was “so methodologically flawed as to be worthless.”
Witte shot back that the newer study by Peterson and his colleagues was a “confusing, tortured effort” to find evidence that choice works.
That study found that students in their third and fourth years in the Milwaukee choice program scored 3 and 5 percentage points higher in reading, on average, than students who applied to the program but were not selected and remained in the public schools. The voucher students averaged 5 and 12 percentage points higher in math, Peterson and his colleagues report. “If similar success could be achieved for all minority students nationwide, it could close the gap separating white and minority test scores by somewhere between one-third and one-half,” the researchers write.
The five annual reports issued by Witte and his colleagues between 1991 and 1995 found no such improvement. Although parents who sent their children to private schools under the plan were more satisfied and more involved than their public school counterparts, the participating students’ test scores neither improved nor differed significantly from those of other Milwaukee students, his reports concluded.
How could two studies of the same program reach such divergent conclusions?
“These are radically different studies,” says Elmore, “and I think they were done for different purposes.”
The most notable difference involves the selection of the control groups against which the choice students were compared. Witte and his colleagues compared data for students in all 12 of the private schools in the program with scores both from Milwaukee public school students as a whole and from the district’s low-income students. They used statistical analysis to account for differences between the choice-program students and the control groups that might have influenced test scores, such as race, income, and sex.
But Peterson and his colleagues contend that some of the differences between the groups were so substantial that even with such controls, those studies “compared apples to oranges.” Their research compared voucher students with children who applied for the program but were not selected. They believe this analysis produced more accurate results.
Students applied each year for a seat in a particular grade in a particular school. If there were more applicants than seats available, students were chosen at random. This, the researchers say, created two ideal experimental groups--the selected and nonselected students.
Theoretically, education researchers say, this is a much stronger design than Witte’s. But, they point out, the new study looked at a far smaller slice of the choice program enrollment than the other research did. By the program’s fourth year, test data in math were available for only 110 selected and nonselected students, out of more than 1,000 children who have participated in the program. The small numbers were caused by the program’s high attrition rate, which averaged 30 percent a year.
The small samples, Elmore says, make it hard to draw broad conclusions from the findings.
The studies also differed in the way they looked at test scores. Witte’s analyses compared changes in test scores for groups of students from one year to the next. But Peterson asserts that such an analysis, which controls for the previous year’s test scores, masks the long-term improvements in those scores that the program was designed to create.
His team, operating on the theory that students need time to adjust to a new learning environment, looked at the performance of individual students over time. They found that choice students did not outperform nonselected students in the program’s first two years but that those who remained in the program at least three years pulled ahead of the control group.
The biggest problem for both teams of researchers was the high attrition rate. The two teams of researchers made different assumptions about the students who left the program. Peterson and his colleagues cite data showing that the students who remained and those who left after two years had similar scores at the end of the second year. They argue that there was little academic difference between those who stayed and those who left.
But Witte maintains that the students who left the program were, on average, those with lower prior test scores. Those who stayed in, he argues, were a select group that skewed the program’s success.
Still, outside experts say the Peterson study is potentially important. “It’s evocative,” says Peter Cookson, director for
educational outreach and innovation at Teachers College, Columbia University. “He may actually be correct. But the study that he conducted is extremely limited, and it’s not a good idea to base large-scale policy decisions on it.”
