School & District Management

Schools Struggle to Use Data to Spark Improvement

By Benjamin Herold — March 20, 2018 | Corrected: March 21, 2018 7 min read

Corrected: A previous version of this article misspelled the name of Menomonee Falls School District. The name of the Carnegie Foundation for the Advancement of Teaching has also been corrected.

Can educational data be used to actually help schools get better?

That’s the hope behind a push to bring to K-12 schools a management philosophy known as “continuous improvement” that has flourished in fields such as health care, manufacturing, and social services.

But experts contend that the K-12 education system’s current data infrastructure, built in response to the federal No Child Left Behind Law and focused primarily on holding schools accountable, looms as a significant barrier.

“New data systems really have to allow [teachers and principals] to say what they want to accomplish, and determine whether they’ve accomplished it,” said Louis Gomez, a professor of education and information studies at the University of California, Los Angeles, and a co-author of the 2015 book, Learning to Improve, How America’s Schools Can Get Better at Getting Better. “Just monitoring how we’re doing is not going to help us get better.”

Examples of what an improvement-based data infrastructure actually looks like in practice are few and far between.

Louis GomezUCLA Professor

Wisconsin’s 4,000-student Menomonee school district is lauded for using a comprehensive approach to steadily improving everything from recess safety to the cleanliness of its classrooms, but educators and administrators there have relied on mostly informal systems for collecting and analyzing information.

The 400,000-student Chicago school district has been at the forefront of efforts to use ‘early-warning’ data to keep high school students on track to graduation and college, but such work typically relies on existing data systems to support a narrowly targeted improvement project. And the New York-based nonprofit New Visions for Public Schools has taken the lead on developing brand new data systems designed to help schools collect and track the kinds of “process data” that continuous-improvement proponents describe as critical, but such tools remain on the fringes of the K-12 sector.

Far more typical are districts like Dallas, where budget woes and leadership churn have hampered efforts to build and use data systems in new ways.

See Also: A Primer on Continuous School Improvement

As a field, education has “placed tremendous emphasis on data for accountability purposes,” said Mark Dunetz, the president of New Visions for Public Schools. “We’ve done a much poorer job at providing practitioners with the tools they need to manage the types of tasks that are the day-to-day bread and butter of schools, and that enable everything else to happen.”

Little Impact on Teaching

The idea of schools making better use of educational data is nothing new.

Back in 2010, the U.S. Department of Education described data-driven decision making as a “national priority,” saying that states and school districts had made significant progress in building a new data infrastructure.

But in a report that year titled “Use of Education Data at the Local Level: From Accountability to Instructional Improvement,” the department also concluded that such work was having little impact on actual classroom teaching.

The reasons?

“School staffs’ perception of barriers to greater use of data include a sense of lack of time, system-usability issues, the perception that the data in the system are not useful, and district policies around curriculum coverage or pacing that prohibit modifying learning time to match student needs,” the report concludes.

Fast forward eight years, and the lingo has changed, but such challenges are still endemic, said Andrew Krumm, the director of learning analytics at Digital Promise, a nonprofit that promotes the use of technology to improve schools.

Andrew KrummData Researcher, Digital Promise

Too often, Krumm and other proponents of continuous improvement say, schools get caught in a cycle in which administrators and policymakers identify a problem; develop a program or intervention to address that problem; roll it out; and try to determine whether or not it worked. The data used to measure success is typically a long-term outcome measure, such as standardized test results. If scores go up, administrators try to scale the program across other schools and districts—often with little consideration for why the program worked, or how conditions on the ground in other schools and districts might vary.

That approach amounts to “implementing fast and learning slow,” Gomez and his co-authors from the Carnegie Foundation for the Advancement of Teaching argue in their 2015 book.

Instead, they argue, schools should be focused on the reverse: implementing improvement strategies more methodically, but learning much more about them as they happen.

Ideally, they say, such a process would entail identifying the problem schools want to fix; developing a theory about how to improve it; and then helping the people closest to the problem—usually teachers, principals, and other school staff—to develop measures of day-to-day progress that are aligned to that theory. Technology tools should help schools monitor three things: whether they’re actually doing what they set out to do, whether it’s making a difference on the measures that educators developed locally, and how such efforts impact the kinds of long-term outcomes measures that are typically used now.

“The most interesting work happens when schools supplement system-wide data with finer-grained, day-in-and-day-out data,” Krumm said in an interview. “We need to be collecting data that is much closer to what we’re actually trying to change.”

Tracking Daily Decisions

That’s generally the approach that New Visions for Public Schools is trying to take with the custom educational software it’s building.

The nonprofit currently provides its data tools to 342 schools in New York City, including 10 charter schools that it directly operates and several dozen district schools for which the group serves as a “comprehensive lead partner.”

The focus is on closely tracking the seemingly mundane decisions that principals, assistant principals, teachers, and guidance counselors have to make each day, then getting them the information they need to make those decisions.

Graduation planning is one big example. Additional functions aim to help counselors match students to internships and other opportunities in the most efficient way possible, or to help administrators track student attendance and participation in after-school activities.

“I think we’ve historically underestimated the complexity of work in schools, where there is a tremendous volume of decisions that need to be made, and the outcomes are often determined by our ability to be consistent,” Dunetz said. “If you don’t know whether a classroom had the right textbooks, or whether the computers worked, or if kids showed up, it’s very difficult to draw any conclusions about whether what you’re doing is working, and why.”

Notably, though, Dunetz said New Visions determined it had to build that kind of software from scratch. Very few software applications allow schools real flexibility to decide what data are most important to them, then collect and analyze that information customized ways.

In theory, at least, the new technologies that have flooded into schools over the past decade could help. Countless digital tools are now part of students’ everyday learning, and many are capable of generating reams of data on everything that a student (or teacher) does.

But much of that software still serves accountability purposes, Krumm said. The experience of the 157,000-student Dallas school system highlights the challenges on the ground to building new data systems to support continuous improvement work.

Back in 2009, the Gates Foundation gave the Dallas district a three-year, $3.8 million grant to strengthen its “college readiness warning system,” modeled after Chicago’s efforts. The aim was to help teachers use data, pulled primarily from existing student-information systems, to “identify student needs, provide appropriate interventions, and ultimately increase college readiness.”

The tool the district developed was “a dashboard pulling up live data” on such indicators as student attendance and access to financial-aid counseling, said Cecilia Oakeley, who started with the district in 2005 and is now an assistant superintendent for evaluation and assessment.

That system took a while to build, Oakeley said. Some schools used it more than others.

And then the grant ran out, the superintendent left, and budget cuts hit.

“The platform went by the wayside,” she said. “When there are switches in administration, some work gets lost in the shuffle.”

Fortunately, Oakeley said, all was not lost. The district had also built a supplemental data system to allow school staff to also examine other relevant information, such as students’ grades in core subject areas. District-level staff still use that system to monitor how many students are on track for graduation at the end of 9th grade. And former superintendent Michael Hinojosa, who led the original push for an early-warning system in Dallas before moving on, is now back leading the district, bringing renewed attention to efforts to get school-level staff to use the information, too.

But for now, at least, that’s still a far cry from the type of continuous-improvement data systems that proponents envision.

“This is a process,” said Gomez of UCLA. “People are coming to understand that all these [data systems we have now] were not created with improvement in mind. This is a first and important step.”

Coverage of continuous-improvement strategies in education is supported in part by a grant from the Bill & Melinda Gates Foundation at www.gatesfoundation.org. Education Week retains sole editorial control over the content of this coverage.
Coverage of continuous-improvement strategies in education is supported in part by a grant from the Bill & Melinda Gates Foundation, at www.gatesfoundation.org. Education Week retains sole editorial control over the content of this coverage.
A version of this article appeared in the March 21, 2018 edition of Education Week as Schools Struggle to Use Data to Get Better

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Professional Development Webinar
Building Leadership Excellence Through Instructional Coaching
Join this webinar for a discussion on instructional coaching and ways you can link your implement or build on your program.
Content provided by Whetstone Education/SchoolMint
Teaching Webinar Tips for Better Hybrid Learning: Ask the Experts What Works
Register and ask your questions about hybrid learning to our expert panel.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Families & the Community Webinar
Family Engagement for Student Success With Dr. Karen Mapp
Register for this free webinar to learn how to empower and engage families for student success featuring Karen L. Mapp.
Content provided by Panorama Education & PowerMyLearning

EdWeek Top School Jobs

Director PEAK Academy Hapeville campus
Hapeville, Georgia, United States
Camelot Education
Technology Product Manager
Philadelphia, Pennsylvania, United States
Camelot Education
2021-2022 Teacher (Districtwide)
Dallas, TX, US
Dallas Independent School District
[2021-2022] Founding Middle School Academic Dean
New York, NY, US
DREAM Charter School

Read Next

School & District Management One District's COVID-19 Journey: A Year of Upheaval and Unexpected Insights
The Everett, Wash., school district was among the first in the nation shuttered by the pandemic. Educators and parents share and reflect.
9 min read
School & District Management The Key to School-Based COVID-19 Testing: Cooperation of Parents and Communities
As schools launch broad testing to track cases of COVID-19, the success of their efforts relies on addressing the concerns of all concerned.
7 min read
Katie Ramirez, left, watches as her mother, Claudia Campos, swabs the mouth of her sister, Hailey, for a COVID-19 test at a testing site in Los Angeles on Dec. 9, 2020.
Katie Ramirez, left, watches as her mother, Claudia Campos, swabs the mouth of her sister, Hailey, for a COVID-19 test at a testing site in Los Angeles.
Jae C. Hong/AP
School & District Management Interactive A Year of COVID-19: What It Looked Like for Schools
This timeline offers a look at how a full year of living and learning during the COVID-19 pandemic unfolded.
Education Week Staff
13 min read
Elementary 1 teacher Melissa Vozar sits outside of Suder Elementary in Chicago to teach a virtual class on Jan. 11, 2021. The Chicago Teachers Union said that its members voted to defy an order to return to the classroom before they are vaccinated against the coronavirus, setting up a showdown with district officials who have said such a move would amount to an illegal strike.
Elementary 1 teacher Melissa Vozar sits outside of Suder Elementary in Chicago to teach a virtual class on Jan. 11, 2021. The Chicago Teachers Union said that its members voted to defy an order to return to the classroom before they are vaccinated against the coronavirus, setting up a showdown with district officials who have said such a move would amount to an illegal strike.
Anthony Vazquez/Chicago Sun-Times via AP
School & District Management Most Principals, District Leaders Predict Their Schools Will Be Fully In-Person This Fall
EdWeek Research Center surveys track the growing trend to get more students back in school buildings as soon as possible.
5 min read
Assistant Principal Janette Van Gelderen, left, welcomes students at Newhall Elementary in Santa Clarita, Calif on Feb. 25, 2021. California's public schools could get $6.6 billion from the state Legislature if they return to in-person instruction by the end of March, according to a new agreement announced Monday, March 1, 2021, between Gov. Gavin Newsom and the state's legislative leaders.
Assistant Principal Janette Van Gelderen, left, welcomes students at Newhall Elementary in Santa Clarita, Calif., last month. California's public schools could get $6.6 billion from the state if they return to in-person instruction by the end of March.
Marcio Jose Sanchez/AP