Compared with their wealthy counterparts, many low-income school districts in Massachusetts are doing more with less when it comes to performance on state exams, a new analysis contends.
The recently released research, which evaluates a wide range of factors that could influence test performance, also found that school spending patterns can affect test scores. And it concludes that the more school districts spend on sports, the worse they do on the exams.
Because it identifies better-than-expected scores among some low-income schools, the study also challenges the belief that demographics are destiny when it comes to performance on the Massachusetts Comprehensive Assessment System, which students in the class of 2003 must pass in English and mathematics to graduate.
“Still near the top [of the performance rankings] are some affluent, high-scoring districts whose performance has been touted in the media on the basis of raw scores,” the report released Feb. 20 states."But the bottom of the list contains some stunning surprises, including several of the most affluent districts in the state.”
The study, “School District Performance Under the MCAS,” is one of the most comprehensive statistical reviews ever of the controversial MCAS exams.
Thomas Ferguson, a professor of political science at the University of Massachusetts, Boston, along with Jie Chen, a statistician there, wrote the report. But the 104-page study has drawn criticism, in part, because its statistical review places some of the state’s most affluent schools at the bottom of its performance rankings.
State Education Commissioner David P. Driscoll, for one, has dismissed the report as using “elaborate formulations” that produced results at odds with what has actually taken place in districts around the state.
“The bottom line is it doesn’t seem to match up with reality,” Mr. Driscoll said. Since the paper looks at MCAS performance from 1998 through 2000, Mr. Driscoll added that the analysis sheds little light on the dramatic improvements many districts made last year.
The MCAS became the centerpiece of reform efforts in Massachusetts after lawmakers passed a sweeping education reform act nine years ago. (“Mass. School Policies Praised as Test Scores Rise,” Oct. 24, 2001.)
The exams have gained strong support from advocates who argue that the MCAS has helped bolster academic expectations for all students. But critics, including Massachusetts’ largest teachers’ union and some students who have boycotted the exams, view the MCAS as a flawed, one-size-fits-all measure of achievement.
‘Value Added’
Among other areas, the authors found that teachers’ and superintendents’ salaries, spending on athletic programs, political races, and state aid all affected district MCAS results.The researchers evaluated every school district in the state by calculating their “grand average” on the exams, which were given to 4th, 8th, and 10th graders in 1998, 1999, and 2000.
Their model also examined districts by using a “value-added” analysis that attempts to estimate the three-year average score that would be predicted for each district purely on the basis of its wealth and demographics. This score is then compared with the district’s actual average to show how much the district over- or under-performed. Using this model, the authors ranked districts and found that some school systems like the Dover- Sherborn district and the Weston public schools, two of the wealthiest in the state, performed well on raw scores, but fared far worse on the MCAS after their economic and demographic advantages were taken into consideration.
In contrast, some less wealthy districts like Lenox, Belmont, and Nauset did far better than expected.
“Here, at last, is a list of high-performing schools whose entries cannot possibly be dismissed as simple reflections of socioeconomic advantage,” the authors write. Mr. Ferguson said in an interview that poverty still consistently remains a “major educational handicap,” but that much of education research has too often ignored other influences that fall outside teacher control.
For example, one of the most intriguing findings of this paper, he said, is that the more a district spends on athletics, the worse their MCAS performance. The researchers found that for each percent a district spends of the amount targeted for athletics in the state’s foundation budget for that district, MCAS scores fall by about a hundredth of a point.
While this may not sound significant, he said, some districts spend more than 300 percent of their targeted athletic spending and could be dragging down their scores three or four points."In terms of effects, this is huge,” Mr. Ferguson said. “The athletic spending point is an overwhelmingly important one because it’s not something under teacher control. If you don’t have a measure for athletic spending you have a major impact you’re not taking into account.”
Range of Factors
Perry P. Davis, the superintendent of the suburban 1,900-student Dover-Sherborn district that finished second-to-last among 226 districts ranked by the value-added research, said he plans to meet with the researchers. Mr. Davis doesn’t understand the authors’ statistical technique, but said he looks forward to learning more about how they evaluated test data. “As a good teacher, if I don’t understand something I want to figure it out,” Mr. Davis said.
The authors defended the value-added formula as a common technique that provides a more sophisticated analysis of data. “This is not a new technique. We didn’t invent this scheme,” Mr. Ferguson said.
Other findings suggest that MCAS scores have been improved moderately by the state’s almost $7 billion investment in K-12 education since 1993. Raising maximum teacher salaries and superintendents’ salaries by $1,000, researchers found, led to slightly higher MCAS scores.
