Assessment

Researchers Find ‘Surprises’ In Mass. Scores

By John Gehring — March 13, 2002 4 min read

Compared with their wealthy counterparts, many low-income school districts in Massachusetts are doing more with less when it comes to performance on state exams, a new analysis contends.

The recently released research, which evaluates a wide range of factors that could influence test performance, also found that school spending patterns can affect test scores. And it concludes that the more school districts spend on sports, the worse they do on the exams.

Because it identifies better-than-expected scores among some low-income schools, the study also challenges the belief that demographics are destiny when it comes to performance on the Massachusetts Comprehensive Assessment System, which students in the class of 2003 must pass in English and mathematics to graduate.

“Still near the top [of the performance rankings] are some affluent, high-scoring districts whose performance has been touted in the media on the basis of raw scores,” the report released Feb. 20 states."But the bottom of the list contains some stunning surprises, including several of the most affluent districts in the state.”

The study, “School District Performance Under the MCAS,” is one of the most comprehensive statistical reviews ever of the controversial MCAS exams.

Thomas Ferguson, a professor of political science at the University of Massachusetts, Boston, along with Jie Chen, a statistician there, wrote the report. But the 104-page study has drawn criticism, in part, because its statistical review places some of the state’s most affluent schools at the bottom of its performance rankings.

State Education Commissioner David P. Driscoll, for one, has dismissed the report as using “elaborate formulations” that produced results at odds with what has actually taken place in districts around the state.

“The bottom line is it doesn’t seem to match up with reality,” Mr. Driscoll said. Since the paper looks at MCAS performance from 1998 through 2000, Mr. Driscoll added that the analysis sheds little light on the dramatic improvements many districts made last year.

The MCAS became the centerpiece of reform efforts in Massachusetts after lawmakers passed a sweeping education reform act nine years ago. (“Mass. School Policies Praised as Test Scores Rise,” Oct. 24, 2001.)

The exams have gained strong support from advocates who argue that the MCAS has helped bolster academic expectations for all students. But critics, including Massachusetts’ largest teachers’ union and some students who have boycotted the exams, view the MCAS as a flawed, one-size-fits-all measure of achievement.

‘Value Added’

Among other areas, the authors found that teachers’ and superintendents’ salaries, spending on athletic programs, political races, and state aid all affected district MCAS results.The researchers evaluated every school district in the state by calculating their “grand average” on the exams, which were given to 4th, 8th, and 10th graders in 1998, 1999, and 2000.

Their model also examined districts by using a “value-added” analysis that attempts to estimate the three-year average score that would be predicted for each district purely on the basis of its wealth and demographics. This score is then compared with the district’s actual average to show how much the district over- or under-performed. Using this model, the authors ranked districts and found that some school systems like the Dover- Sherborn district and the Weston public schools, two of the wealthiest in the state, performed well on raw scores, but fared far worse on the MCAS after their economic and demographic advantages were taken into consideration.

In contrast, some less wealthy districts like Lenox, Belmont, and Nauset did far better than expected.

“Here, at last, is a list of high-performing schools whose entries cannot possibly be dismissed as simple reflections of socioeconomic advantage,” the authors write. Mr. Ferguson said in an interview that poverty still consistently remains a “major educational handicap,” but that much of education research has too often ignored other influences that fall outside teacher control.

For example, one of the most intriguing findings of this paper, he said, is that the more a district spends on athletics, the worse their MCAS performance. The researchers found that for each percent a district spends of the amount targeted for athletics in the state’s foundation budget for that district, MCAS scores fall by about a hundredth of a point.

While this may not sound significant, he said, some districts spend more than 300 percent of their targeted athletic spending and could be dragging down their scores three or four points."In terms of effects, this is huge,” Mr. Ferguson said. “The athletic spending point is an overwhelmingly important one because it’s not something under teacher control. If you don’t have a measure for athletic spending you have a major impact you’re not taking into account.”

Range of Factors

Perry P. Davis, the superintendent of the suburban 1,900-student Dover-Sherborn district that finished second-to-last among 226 districts ranked by the value-added research, said he plans to meet with the researchers. Mr. Davis doesn’t understand the authors’ statistical technique, but said he looks forward to learning more about how they evaluated test data. “As a good teacher, if I don’t understand something I want to figure it out,” Mr. Davis said.

The authors defended the value-added formula as a common technique that provides a more sophisticated analysis of data. “This is not a new technique. We didn’t invent this scheme,” Mr. Ferguson said.

Other findings suggest that MCAS scores have been improved moderately by the state’s almost $7 billion investment in K-12 education since 1993. Raising maximum teacher salaries and superintendents’ salaries by $1,000, researchers found, led to slightly higher MCAS scores.

Related Tags:

A version of this article appeared in the March 13, 2002 edition of Education Week as Researchers Find ‘Surprises’ In Mass. Scores

Events

School & District Management Live Event EdWeek Leadership Symposium
Education Week's Premier Leadership Event for K12 School & District Leaders.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Law & Courts Webinar
The Future of Criminal Justice Reform: A Sphere Education Initiative Conversation
America’s criminal justice system is in crisis and calls for reform are dominating the national debate. Join Cato’s Sphere Education Initiative and Education Week for a webinar on criminal justice and policing featuring the nation’s
Content provided by Cato Institute
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Equity, Care and Connection: New SEL Tools and Practices to Support Students and Adults
As school districts plan to welcome students back into buildings for the upcoming school year, this is the perfect time to take a hard look at both our practices and our systems to build a
Content provided by Panorama Education

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Assessment Opinion The National Assessment Governing Board’s Troubling Gag Order
NAGB's recently released restrictions on how its board members can communicate set a troubling precedent.
3 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Whitepaper
INVALSI Addresses Italy’s COVID-19 Learning Loss
Find out how INVALSI worked with TAO to develop a plan of action that can serve as a model for other education leaders grappling with the...
Content provided by TAO by Open Assessment Technologies
Assessment Biden Administration's Level of Tolerance for Cutting Standardized Tests Comes Into Focus
A distinction has grown between states having to make tests available, and districts deciding it's not practical to make students take them.
8 min read
Image of a test sheet.
sengchoy/iStock/Getty
Assessment Opinion Alternatives to Standardized Tests During a Pandemic Year
Three educators suggest alternatives to federally mandated standardized testing during this year undercut by COVID-19.
7 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
iStock/Getty