School & District Management

Study Gives First Look at Data’s Impact on Districts

By Jaclyn Zubrzycki — February 21, 2012 5 min read
  • Save to favorites
  • Print

A new study from the University of Wisconsin-Madison is among the first to examine the effectiveness of a data-driven effort to improve schooling on a large scale, and it’s good news for data advocates: Even the first steps of data-driven instruction seem to have some positive impact on school districts’ test scores.

The report focuses on 59 districts in Alabama, Arizona, Indiana, Ohio, Mississippi, Tennessee, and Pennsylvania, examining their mathematics and reading results on state standardized tests after the first year of a three-year initiative conducted by the Center for Data-Driven Reform in Education at Johns Hopkins University, in Baltimore.

The study is drawing attention as much for its design as its results. While most randomized studies in education are conducted at the individual, school, or class level, researchers for this study randomly assigned entire districts either to experimental or control groups, which allowed a controlled comparison of districtwide changes in student performance on standardized tests.

“Given how central data is to a lot of reforms that school districts in states across the country are doing, with data emerging from No Child Left Behind and data being generated by the current [federal] department of education, it’s reasonable to ask the question, do the data actually matter?” said Michael Casserly, the executive director of the Washington-based Council of the Great City Schools. This study is one of the first to show that those efforts do work on a large scale, he said.

Districts’ Scores Improve

States began the program in three successive waves, starting in 2005. In each wave, researchers gave one group of districts benchmark assessments tailored to their state examinations and trained school administrators in interpreting and using data to identify areas for instructional improvement. The second group received those services the next year and served as a control group in the first year of the study. Districts that received the assessments had greater gains on their states’ reading and math tests than districts that had not yet begun the experiment.

On average, students from the data-driven-reform districts outperformed their control-group counterparts by approximately8 percentile points in math and 5 percentile points in reading, according to Geoffrey D. Borman, a professor of sociology and education at the University of Wisconsin-Madison and one of the study’s authors. Researchers also noted correlations between districts’ scores and their percentage of students eligible for free- or reduced-price lunch, and found that participating in the CDDRE study was “comparable to reducing school-level free- or reduced-price lunch eligibility by approximately 35 to 60 percentage points,” Mr. Borman said.

While most states are already implementing data-driven reform, research on its impact on student learning, especially at the district level, is sparse, researchers said. The CDDRE study is “important because of its size and its breadth,” said Martin Orland, the director of evaluation and policy research at WestEd, a San Francisco-based research firm. “It’s the largest study that I’m aware of that looks at the issue of data-driven decisionmaking, and it’s enough to be intriguing that there may be supportive evidence for what scholarship has hypothesized and what people are acting on,” he said.

Diverse District Mix

The districts assigned were generally low-performing, but the 549 schools represented were geographically and demographically diverse, Mr. Orland said, which makes the results more likely to be widely relevant.

Looking at reform on a larger scale is important, said Robert Slavin, the director of the data-driven-reform center, whose efforts are documented in the report. Mr. Slavin also writes an independent blog on education research that is hosted on Education Week‘s website. “People quite legitimately say, ‘So what? You can do something in a small number of schools,’ ” Mr. Slavin said. “But this is at such a large scale.”

More Than Data

At the same time, “a study of this breadth raises more questions than it answers,” said WestEd’s Mr. Orland. “What would be ideal would be if you also had some survey information. Were teachers doing things differently? What were they doing that was different? It cries out for further investigation.”

Indeed, the researchers in the new study write that while improvement in test scores could be the result of teachers’ effective use of data, other research frequently shows that teachers do not know how to use data effectively. The improvement could instead demonstrate what Mr. Borman referred to as “the testing effect"—students tend to score better on tests after having practiced for them.

Paige Kowalski, the director of state policy initiatives at the Washington-based Data Quality Campaign, which promotes data-based instruction and reform, said her organization had found that “by and large, teachers don’t yet have the skills and capacity to take results from interim assessments [like those used in this study] and change their own instruction.”

Patte Barthe, the director of the National School Boards Association’s Center for Public Education, said her organization had already been supporting data-driven reform and was encouraged by the study. “What we’re thinking is a good idea is proving to be a good idea,” she said. But, she said, “now that we have tools available, we have to think about ways to help educators, school boards, and parents make use of these tools.”

Results from subsequent years of the program, during which some of the districts adopted various interventions to address areas of need, will be published in an upcoming study, said Mr. Slavin, of CDDRE. That publication will delve into which interventions were used, and indicates that districts’ scores improved more dramatically after they began using interventions to address the problem areas revealed by the data than when they had access only to the benchmark data.

“Benchmarks are useful, but they’re only a part of the process,” Mr. Slavin said.

Which Data Matter?

The Data Quality Campaign’s Ms. Kowalski said that advocates of data-driven reform were moving to use data about attendance, school climate, and other factors outside of test scores.

“We know that the benchmark assessments are just one kind of data,” she said.

Mr. Orland said focusing on improvements in state test scores “does run the danger of equating the test with real learning.” Some of the most effective reforms use additional data like school climate surveys and attendance, he said.

But especially as many schools lag in implementing holistic reforms, Mr. Borman, the study’s author, said “so many districts have been adopting quarterly benchmark assessments that [how they impact student achievement] is an important question in itself.”

Discovering that benchmark assessments have an impact gives a first glimpse at how to “move the needle forward” for student achievement, said Mr. Casserly. “The research is starting to get pretty exciting in terms of informing us what things work and why.”

A version of this article appeared in the February 22, 2012 edition of Education Week as Study Offers First Glimpse of Data’s Impact on Districts

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Law & Courts Webinar
Future of the First Amendment:Exploring Trends in High School Students’ Views of Free Speech
Learn how educators are navigating student free speech issues and addressing controversial topics like gender and race in the classroom.
Content provided by The John S. and James L. Knight Foundation
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Start Strong With Solid SEL Implementation: Success Strategies for the New School Year
Join Satchel Pulse to learn why implementing a solid SEL program at the beginning of the year will deliver maximum impact to your students.
Content provided by Satchel Pulse
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Webinar
Real-World Problem Solving: How Invention Education Drives Student Learning
Hear from student inventors and K-12 teachers about how invention education enhances learning, opens minds, and preps students for the future.
Content provided by The Lemelson Foundation

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Schools Prefer Cheaper Ventilation Options to Curb COVID: Why They Should Consider Upgrading
Most schools are opening windows and hosting class outdoors rather than investing in costlier, more-effective measures.
2 min read
Students from PS 11 Elementary School participate in art projects and interactive activities, during an after-school outdoor program held in the High Line park in New York, NY, October 21, 2020.
Students from PS 11 Elementary School participate in art projects and interactive activities during an after-school outdoor program in New York City in 2020. Many schools are opting for outdoor classes and other-low cost measures to maintain healthy air quality during the pandemic.
Anthony Behar/Sipa via AP Images
School & District Management Hour by Busy Hour: What a Principal's Day Actually Looks Like
From the time they wake up until they set the alarm at night, school leaders juggle the routine, the unexpected, and the downright bizarre.
Left, Principal Michael C. Brown talks on a radio at Winters Mill High School in Westminster, Md., on May 17, 2022. Right, Boone Elementary School principal Manuela Haberer directs students and parents in the pick-up line at the conclusion of the school day on May 19, 2022 in San Antonio, Texas.
Left, Principal Michael C. Brown talks on a radio at Winters Mill High School in Westminster, Md., on May 17, 2022. Right, Boone Elementary School principal Manuela Haberer directs students and parents in the pick-up line at the conclusion of the school day on May 19, 2022 in San Antonio, Texas.
From left, Steve Ruark and Lisa Krantz for Education Week
School & District Management Photos What School Leadership Looks Like: A Day in the Life of a Principal
A look at a typical day for one elementary school principal in Texas and a high school principal in Maryland.
1 min read
Principal Michael C. Brown, from left, talks to seniors Brady D’Anthony, 18, and Sydney Dryden, 17, at Winters Mill High School in Westminster, Md., Tuesday, May 17, 2022.
Principal Michael C. Brown, from left, talks to seniors Brady D’Anthony, 18, and Sydney Dryden, 17, at Winters Mill High School in Westminster, Md., Tuesday, May 17, 2022.
Steve Ruark for Education Week
School & District Management Schools Can Access Tons of Money for Electric Buses. Will They Use It?
Electric buses are growing more appealing as fuel prices rise, but some districts remain wary of the cost and logistics.
5 min read
Stockton Unified School District's new electric bus fleet reduces over 120,000 pounds of carbon emissions and leverages The Mobility House's smart charging and energy management system.
The new electric bus fleet at California's Stockton Unified School District is projected to reduce the district's carbon emissions.
Business Wire via AP