Assessment

Researchers Find ‘Surprises’ In Mass. Scores

By John Gehring — March 13, 2002 4 min read
  • Save to favorites
  • Print

Compared with their wealthy counterparts, many low-income school districts in Massachusetts are doing more with less when it comes to performance on state exams, a new analysis contends.

The recently released research, which evaluates a wide range of factors that could influence test performance, also found that school spending patterns can affect test scores. And it concludes that the more school districts spend on sports, the worse they do on the exams.

Because it identifies better-than-expected scores among some low-income schools, the study also challenges the belief that demographics are destiny when it comes to performance on the Massachusetts Comprehensive Assessment System, which students in the class of 2003 must pass in English and mathematics to graduate.

“Still near the top [of the performance rankings] are some affluent, high-scoring districts whose performance has been touted in the media on the basis of raw scores,” the report released Feb. 20 states."But the bottom of the list contains some stunning surprises, including several of the most affluent districts in the state.”

The study, “School District Performance Under the MCAS,” is one of the most comprehensive statistical reviews ever of the controversial MCAS exams.

Thomas Ferguson, a professor of political science at the University of Massachusetts, Boston, along with Jie Chen, a statistician there, wrote the report. But the 104-page study has drawn criticism, in part, because its statistical review places some of the state’s most affluent schools at the bottom of its performance rankings.

State Education Commissioner David P. Driscoll, for one, has dismissed the report as using “elaborate formulations” that produced results at odds with what has actually taken place in districts around the state.

“The bottom line is it doesn’t seem to match up with reality,” Mr. Driscoll said. Since the paper looks at MCAS performance from 1998 through 2000, Mr. Driscoll added that the analysis sheds little light on the dramatic improvements many districts made last year.

The MCAS became the centerpiece of reform efforts in Massachusetts after lawmakers passed a sweeping education reform act nine years ago. (“Mass. School Policies Praised as Test Scores Rise,” Oct. 24, 2001.)

The exams have gained strong support from advocates who argue that the MCAS has helped bolster academic expectations for all students. But critics, including Massachusetts’ largest teachers’ union and some students who have boycotted the exams, view the MCAS as a flawed, one-size-fits-all measure of achievement.

‘Value Added’

Among other areas, the authors found that teachers’ and superintendents’ salaries, spending on athletic programs, political races, and state aid all affected district MCAS results.The researchers evaluated every school district in the state by calculating their “grand average” on the exams, which were given to 4th, 8th, and 10th graders in 1998, 1999, and 2000.

Their model also examined districts by using a “value-added” analysis that attempts to estimate the three-year average score that would be predicted for each district purely on the basis of its wealth and demographics. This score is then compared with the district’s actual average to show how much the district over- or under-performed. Using this model, the authors ranked districts and found that some school systems like the Dover- Sherborn district and the Weston public schools, two of the wealthiest in the state, performed well on raw scores, but fared far worse on the MCAS after their economic and demographic advantages were taken into consideration.

In contrast, some less wealthy districts like Lenox, Belmont, and Nauset did far better than expected.

“Here, at last, is a list of high-performing schools whose entries cannot possibly be dismissed as simple reflections of socioeconomic advantage,” the authors write. Mr. Ferguson said in an interview that poverty still consistently remains a “major educational handicap,” but that much of education research has too often ignored other influences that fall outside teacher control.

For example, one of the most intriguing findings of this paper, he said, is that the more a district spends on athletics, the worse their MCAS performance. The researchers found that for each percent a district spends of the amount targeted for athletics in the state’s foundation budget for that district, MCAS scores fall by about a hundredth of a point.

While this may not sound significant, he said, some districts spend more than 300 percent of their targeted athletic spending and could be dragging down their scores three or four points."In terms of effects, this is huge,” Mr. Ferguson said. “The athletic spending point is an overwhelmingly important one because it’s not something under teacher control. If you don’t have a measure for athletic spending you have a major impact you’re not taking into account.”

Range of Factors

Perry P. Davis, the superintendent of the suburban 1,900-student Dover-Sherborn district that finished second-to-last among 226 districts ranked by the value-added research, said he plans to meet with the researchers. Mr. Davis doesn’t understand the authors’ statistical technique, but said he looks forward to learning more about how they evaluated test data. “As a good teacher, if I don’t understand something I want to figure it out,” Mr. Davis said.

The authors defended the value-added formula as a common technique that provides a more sophisticated analysis of data. “This is not a new technique. We didn’t invent this scheme,” Mr. Ferguson said.

Other findings suggest that MCAS scores have been improved moderately by the state’s almost $7 billion investment in K-12 education since 1993. Raising maximum teacher salaries and superintendents’ salaries by $1,000, researchers found, led to slightly higher MCAS scores.

Related Tags:

A version of this article appeared in the March 13, 2002 edition of Education Week as Researchers Find ‘Surprises’ In Mass. Scores

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Webinar
Reflections on Evidence-Based Grading Practices: What We Learned for Next Year
Get real insights on evidence-based grading from K-12 leaders.
Content provided by Otus
Classroom Technology K-12 Essentials Forum How AI Use Is Expanding in K-12 Schools
Join this free virtual event to explore how AI technology is—and is not—improving K-12 teaching and learning.
Mathematics Webinar How to Build Students’ Confidence in Math
Learn practical tips to build confident mathematicians in our webinar.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment Interactive How Teachers Approach Grading, in Charts
Teachers weigh multiple factors, including subjective perceptions, when selecting grades, beyond just assignments and test scores.
3 min read
Assessment Why Some Schools Are Ditching Class Rank and Weighted GPAs
Educators wonder whether it is time to revisit class rank and weighted GPAs.
8 min read
Grading and assessment SR
Robert Neubecker for Education Week
Assessment Grading for Equity: Inside One District’s Big Policy Shift
Districts have been shifting grading to strictly assess student learning without add-ons such as extra credit.
8 min read
Image of students lined up
Robert Neubecker for Education Week
Assessment What Are Grades Really For? What Research Says About 4 Common Answers
Differing opinions about the purpose of grades are at the heart of the grading debate.
6 min read
Image of students holding up transparent transcripts.
Robert Neubecker for Education Week