Assessment

Researchers Find ‘Surprises’ In Mass. Scores

By John Gehring — March 13, 2002 4 min read
  • Save to favorites
  • Print

Compared with their wealthy counterparts, many low-income school districts in Massachusetts are doing more with less when it comes to performance on state exams, a new analysis contends.

The recently released research, which evaluates a wide range of factors that could influence test performance, also found that school spending patterns can affect test scores. And it concludes that the more school districts spend on sports, the worse they do on the exams.

Because it identifies better-than-expected scores among some low-income schools, the study also challenges the belief that demographics are destiny when it comes to performance on the Massachusetts Comprehensive Assessment System, which students in the class of 2003 must pass in English and mathematics to graduate.

“Still near the top [of the performance rankings] are some affluent, high-scoring districts whose performance has been touted in the media on the basis of raw scores,” the report released Feb. 20 states."But the bottom of the list contains some stunning surprises, including several of the most affluent districts in the state.”

The study, “School District Performance Under the MCAS,” is one of the most comprehensive statistical reviews ever of the controversial MCAS exams.

Thomas Ferguson, a professor of political science at the University of Massachusetts, Boston, along with Jie Chen, a statistician there, wrote the report. But the 104-page study has drawn criticism, in part, because its statistical review places some of the state’s most affluent schools at the bottom of its performance rankings.

State Education Commissioner David P. Driscoll, for one, has dismissed the report as using “elaborate formulations” that produced results at odds with what has actually taken place in districts around the state.

“The bottom line is it doesn’t seem to match up with reality,” Mr. Driscoll said. Since the paper looks at MCAS performance from 1998 through 2000, Mr. Driscoll added that the analysis sheds little light on the dramatic improvements many districts made last year.

The MCAS became the centerpiece of reform efforts in Massachusetts after lawmakers passed a sweeping education reform act nine years ago. (“Mass. School Policies Praised as Test Scores Rise,” Oct. 24, 2001.)

The exams have gained strong support from advocates who argue that the MCAS has helped bolster academic expectations for all students. But critics, including Massachusetts’ largest teachers’ union and some students who have boycotted the exams, view the MCAS as a flawed, one-size-fits-all measure of achievement.

‘Value Added’

Among other areas, the authors found that teachers’ and superintendents’ salaries, spending on athletic programs, political races, and state aid all affected district MCAS results.The researchers evaluated every school district in the state by calculating their “grand average” on the exams, which were given to 4th, 8th, and 10th graders in 1998, 1999, and 2000.

Their model also examined districts by using a “value-added” analysis that attempts to estimate the three-year average score that would be predicted for each district purely on the basis of its wealth and demographics. This score is then compared with the district’s actual average to show how much the district over- or under-performed. Using this model, the authors ranked districts and found that some school systems like the Dover- Sherborn district and the Weston public schools, two of the wealthiest in the state, performed well on raw scores, but fared far worse on the MCAS after their economic and demographic advantages were taken into consideration.

In contrast, some less wealthy districts like Lenox, Belmont, and Nauset did far better than expected.

“Here, at last, is a list of high-performing schools whose entries cannot possibly be dismissed as simple reflections of socioeconomic advantage,” the authors write. Mr. Ferguson said in an interview that poverty still consistently remains a “major educational handicap,” but that much of education research has too often ignored other influences that fall outside teacher control.

For example, one of the most intriguing findings of this paper, he said, is that the more a district spends on athletics, the worse their MCAS performance. The researchers found that for each percent a district spends of the amount targeted for athletics in the state’s foundation budget for that district, MCAS scores fall by about a hundredth of a point.

While this may not sound significant, he said, some districts spend more than 300 percent of their targeted athletic spending and could be dragging down their scores three or four points."In terms of effects, this is huge,” Mr. Ferguson said. “The athletic spending point is an overwhelmingly important one because it’s not something under teacher control. If you don’t have a measure for athletic spending you have a major impact you’re not taking into account.”

Range of Factors

Perry P. Davis, the superintendent of the suburban 1,900-student Dover-Sherborn district that finished second-to-last among 226 districts ranked by the value-added research, said he plans to meet with the researchers. Mr. Davis doesn’t understand the authors’ statistical technique, but said he looks forward to learning more about how they evaluated test data. “As a good teacher, if I don’t understand something I want to figure it out,” Mr. Davis said.

The authors defended the value-added formula as a common technique that provides a more sophisticated analysis of data. “This is not a new technique. We didn’t invent this scheme,” Mr. Ferguson said.

Other findings suggest that MCAS scores have been improved moderately by the state’s almost $7 billion investment in K-12 education since 1993. Raising maximum teacher salaries and superintendents’ salaries by $1,000, researchers found, led to slightly higher MCAS scores.

Related Tags:

A version of this article appeared in the March 13, 2002 edition of Education Week as Researchers Find ‘Surprises’ In Mass. Scores

Events

Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
Ed-Tech Policy Webinar Artificial Intelligence in Practice: Building a Roadmap for AI Use in Schools
AI in education: game-changer or classroom chaos? Join our webinar & learn how to navigate this evolving tech responsibly.
Education Webinar Developing and Executing Impactful Research Campaigns to Fuel Your Ed Marketing Strategy 
Develop impactful research campaigns to fuel your marketing. Join the EdWeek Research Center for a webinar with actionable take-aways for companies who sell to K-12 districts.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment The 5 Burning Questions for Districts on Grading Reforms
As districts rethink grading policies, they consider the purpose of grades and how to make them more reliable measures of learning.
5 min read
Grading reform lead art
Illustration by Laura Baker/Education Week with E+ and iStock/Getty
Assessment As They Revamp Grading, Districts Try to Improve Consistency, Prevent Inflation
Districts have embraced bold changes to make grading systems more consistent, but some say they've inflated grades and sent mixed signals.
10 min read
Close crop of a teacher's hands grading a stack of papers with a red marker.
E+
Assessment Opinion What's the Best Way to Grade Students? Teachers Weigh In
There are many ways to make grading a better, more productive experience for students. Here are a few.
14 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
iStock/Getty
Assessment Spotlight Spotlight on Assessment
This Spotlight will help you evaluate effective ways to offer students feedback, learn how to improve assessments for ELs, and more.