School & District Management

Study Gives First Look at Data’s Impact on Districts

By Jaclyn Zubrzycki — February 21, 2012 5 min read
  • Save to favorites
  • Print

A new study from the University of Wisconsin-Madison is among the first to examine the effectiveness of a data-driven effort to improve schooling on a large scale, and it’s good news for data advocates: Even the first steps of data-driven instruction seem to have some positive impact on school districts’ test scores.

The report focuses on 59 districts in Alabama, Arizona, Indiana, Ohio, Mississippi, Tennessee, and Pennsylvania, examining their mathematics and reading results on state standardized tests after the first year of a three-year initiative conducted by the Center for Data-Driven Reform in Education at Johns Hopkins University, in Baltimore.

The study is drawing attention as much for its design as its results. While most randomized studies in education are conducted at the individual, school, or class level, researchers for this study randomly assigned entire districts either to experimental or control groups, which allowed a controlled comparison of districtwide changes in student performance on standardized tests.

“Given how central data is to a lot of reforms that school districts in states across the country are doing, with data emerging from No Child Left Behind and data being generated by the current [federal] department of education, it’s reasonable to ask the question, do the data actually matter?” said Michael Casserly, the executive director of the Washington-based Council of the Great City Schools. This study is one of the first to show that those efforts do work on a large scale, he said.

Districts’ Scores Improve

States began the program in three successive waves, starting in 2005. In each wave, researchers gave one group of districts benchmark assessments tailored to their state examinations and trained school administrators in interpreting and using data to identify areas for instructional improvement. The second group received those services the next year and served as a control group in the first year of the study. Districts that received the assessments had greater gains on their states’ reading and math tests than districts that had not yet begun the experiment.

On average, students from the data-driven-reform districts outperformed their control-group counterparts by approximately8 percentile points in math and 5 percentile points in reading, according to Geoffrey D. Borman, a professor of sociology and education at the University of Wisconsin-Madison and one of the study’s authors. Researchers also noted correlations between districts’ scores and their percentage of students eligible for free- or reduced-price lunch, and found that participating in the CDDRE study was “comparable to reducing school-level free- or reduced-price lunch eligibility by approximately 35 to 60 percentage points,” Mr. Borman said.

While most states are already implementing data-driven reform, research on its impact on student learning, especially at the district level, is sparse, researchers said. The CDDRE study is “important because of its size and its breadth,” said Martin Orland, the director of evaluation and policy research at WestEd, a San Francisco-based research firm. “It’s the largest study that I’m aware of that looks at the issue of data-driven decisionmaking, and it’s enough to be intriguing that there may be supportive evidence for what scholarship has hypothesized and what people are acting on,” he said.

Diverse District Mix

The districts assigned were generally low-performing, but the 549 schools represented were geographically and demographically diverse, Mr. Orland said, which makes the results more likely to be widely relevant.

Looking at reform on a larger scale is important, said Robert Slavin, the director of the data-driven-reform center, whose efforts are documented in the report. Mr. Slavin also writes an independent blog on education research that is hosted on Education Week‘s website. “People quite legitimately say, ‘So what? You can do something in a small number of schools,’ ” Mr. Slavin said. “But this is at such a large scale.”

More Than Data

At the same time, “a study of this breadth raises more questions than it answers,” said WestEd’s Mr. Orland. “What would be ideal would be if you also had some survey information. Were teachers doing things differently? What were they doing that was different? It cries out for further investigation.”

Indeed, the researchers in the new study write that while improvement in test scores could be the result of teachers’ effective use of data, other research frequently shows that teachers do not know how to use data effectively. The improvement could instead demonstrate what Mr. Borman referred to as “the testing effect"—students tend to score better on tests after having practiced for them.

Paige Kowalski, the director of state policy initiatives at the Washington-based Data Quality Campaign, which promotes data-based instruction and reform, said her organization had found that “by and large, teachers don’t yet have the skills and capacity to take results from interim assessments [like those used in this study] and change their own instruction.”

Patte Barthe, the director of the National School Boards Association’s Center for Public Education, said her organization had already been supporting data-driven reform and was encouraged by the study. “What we’re thinking is a good idea is proving to be a good idea,” she said. But, she said, “now that we have tools available, we have to think about ways to help educators, school boards, and parents make use of these tools.”

Results from subsequent years of the program, during which some of the districts adopted various interventions to address areas of need, will be published in an upcoming study, said Mr. Slavin, of CDDRE. That publication will delve into which interventions were used, and indicates that districts’ scores improved more dramatically after they began using interventions to address the problem areas revealed by the data than when they had access only to the benchmark data.

“Benchmarks are useful, but they’re only a part of the process,” Mr. Slavin said.

Which Data Matter?

The Data Quality Campaign’s Ms. Kowalski said that advocates of data-driven reform were moving to use data about attendance, school climate, and other factors outside of test scores.

“We know that the benchmark assessments are just one kind of data,” she said.

Mr. Orland said focusing on improvements in state test scores “does run the danger of equating the test with real learning.” Some of the most effective reforms use additional data like school climate surveys and attendance, he said.

But especially as many schools lag in implementing holistic reforms, Mr. Borman, the study’s author, said “so many districts have been adopting quarterly benchmark assessments that [how they impact student achievement] is an important question in itself.”

Discovering that benchmark assessments have an impact gives a first glimpse at how to “move the needle forward” for student achievement, said Mr. Casserly. “The research is starting to get pretty exciting in terms of informing us what things work and why.”

A version of this article appeared in the February 22, 2012 edition of Education Week as Study Offers First Glimpse of Data’s Impact on Districts

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Webinar
Strategies for Incorporating SEL into Curriculum
Empower students to thrive. Learn how to integrate powerful social-emotional learning (SEL) strategies into the classroom.
Content provided by Be GLAD
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Leadership in Education: Building Collaborative Teams and Driving Innovation
Learn strategies to build strong teams, foster innovation, & drive student success.
Content provided by Follett Learning
School & District Management K-12 Essentials Forum Principals, Lead Stronger in the New School Year
Join this free virtual event for a deep dive on the skills and motivation you need to put your best foot forward in the new year.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management What the Research Says 5 Things Schools Can Do This Summer to Improve Student Attendance Next Year
Schools can get a jump on student attendance during the school year by using data, leveraging summer programs, and connecting with families.
6 min read
Julian Gresham, 12, left, works in a group to program a Bee-Bot while in their fifth grade summer school class Monday, June 14, 2021, at Goliad Elementary School. Bee-bots and are new to Ector County Independent School District and help to teach students basic programming skills like sequencing, estimation and problem-solving.
Julian Gresham, 12, left, works on a robotics programming activity in a 5th-grade summer school class June 14, 2021, at Goliad Elementary School in Ector County, Texas. Active summer programs may improve students' attendance during the school year.
Jacob Ford/Odessa American via AP
School & District Management Grad Rates Soared at a School Few Wanted to Attend. How It Happened
Leaders at this Florida high school have "learned to be flexible" to improve graduation rates.
8 min read
Student hanging on a tearing graduate cap tassel
DigitalVision Vectors/Getty
School & District Management Opinion Don’t Just Listen to the Loudest Voices: Resources for Ed. Leaders
These resources can help school and district leaders communicate with their communities.
Jennifer Perry Cheatham & Jenny Portillo-Nacu
5 min read
A pair of hands type on a blank slate of keys that are either falling apart or coming together on a bed of sharpened pencils.  Leadership resources.
Raul Arias for Education Week
School & District Management The Harm of School Closures Can Last a Lifetime, New Research Shows
The short-term effects on students when their schools close have been well documented. New research examines the long-term impact.
5 min read
Desks and chairs are stacked in an empty classroom after the permanent closure of Queen of the Rosary Catholic Academy in Brooklyn borough of New York on Aug. 6, 2020.
Desks and chairs are stacked in an empty classroom after the permanent closure of Queen of the Rosary Catholic Academy in Brooklyn borough of New York on Aug. 6, 2020. A new study examines the long-term effects on students whose schools close.
Jessie Wardarski/AP