Opinion
School & District Management Opinion

We Must Raise the Bar for Evidence in Education

By Carly Robinson & Todd Rogers — October 30, 2019 5 min read
BRIC ARCHIVE

Those looking for what works in education will find no shortage of advice. Educators, hoping to improve student outcomes, eagerly embrace recommendations telling them to “Cater to each child’s learning style!” and “Give students awards for positive behaviors!” But many such intuitive, popular “best practices” may not, in fact, be what is best for students, even though their proponents stamp them “evidence based.”

Educators who prioritize evidence-based practices are fighting an uphill battle; the standards of proof for what constitutes “evidence” in schools—and education more widely—are often exceedingly low. For instance, the popular notions that we should be teaching to students’ learning styles or providing students with attendance awards were both rooted in observational evidence. Both practices have now been debunked, but over 75 percent of educators still endorse learning styles, and many schools say they use awards to recognize excellent student attendance.

We should not be surprised when people have incorrect notions on what research says works—education research is littered with published papers on a range of practical topics that either do not replicate previous findings or suggest massively inflated effects.

Educational policymakers and practitioners need to understand how study designs and research practices influence the reproducibility and credibility of a study’s findings. This is easier said than done, but there are a couple of initial indicators that suggest a research finding is “real” and worth implementing.

First, to disentangle whether a practice causes improvement or is merely associated with it, we need to use research methods that can reliably identify causal relationships. And the best way to determine whether a practice causes an outcome is to conduct a randomized controlled trial (or “RCT,” meaning participants were randomly assigned to being exposed to the practice under study or not being exposed to it).

The standards of proof for what constitutes 'evidence' in schools—and education more widely—are often exceedingly low."

Second, policymakers and practitioners evaluating research studies should have more confidence in studies where the same findings have been observed multiple times in different settings with large samples. Many educational practices are based on single research studies with small sample sizes. We can learn from small, one-off studies. But when it comes to adopting practices, we recommend those that have been evaluated in studies with large sample sizes and reproducible results.

Finally, we can have much more faith in a study’s findings when they are preregistered. That is, researchers publicly post exactly what their hypotheses are and exactly how they will evaluate each one before they have examined their data. This helps limit flexible research practices, making it less likely that researchers will find statistically significant results by chance.

Some will lament that large RCTs are too expensive, slow, or difficult to implement, and that preregistering studies is not feasible because educational research is messy and unpredictable. Yet several studies conducted in the past few years prove that changes in educational practice evaluated by large-scale, reproducible, preregistered RCTs exist and can be used to inform work on the ground.

In our own work, we have spent the last six years studying how to reduce student absenteeism. Through two large-scale, preregistered RCTs (one with more than 28,000 K-12 students and another with almost 11,000 K-5 students), our research team found that sending mailings to parents several times over the course of the school year with personalized attendance information that dynamically targets key parental misbeliefs consistently reduces chronic absenteeism 10 percent to 15 percent. This research led to the creation of InClassToday, a program that partners with districts around the country to help them reduce student absenteeism by implementing this research-backed intervention.

Another practice that educators can—and, strong evidence suggests, should—take up comes from a study conducted by Peter Bergman and Eric Chan. They randomly assigned parents of more than 1,000 middle and high school students in 22 schools to receive automated, frequent information via text message about their child’s missed assignments and grades. The study, which has now been replicated multiple times, found that this strategy led to a 28 percent reduction in course failures, a 12 percent increase in specific class attendance, and increased student retention by 1.5 percentage points. Providing parents with information that helps them monitor their child’s academic progress can have meaningful impacts on student success.

Finally, a study of learning mindsets led by David Yeager explored for whom growth mindset interventions are most effective. In a nationally representative sample of more than 12,000 students, the study found that adolescents assigned to complete a “growth mindset” intervention—which taught that intellectual abilities can be developed—earned higher GPAs (a modest, but real 0.05-grade points) in core classes at the end of the 9th grade. The authors preregistered that they predicted the intervention would most help low-achieving students. Consistent with this, they found that low-achieving students showed larger effects with higher GPAs (0.10-grade points) in core classes at the end of the 9th grade and were 11 percentage points less likely to get a D or F average in one of these classes. This large-scale and preregistered study replicating prior findings provides evidence that mindset interventions can improve outcomes for struggling students.

Each of these relatively low-cost and easy-to-implement interventions have modest but real impacts on student outcomes. Holding educational research to greater standards of evidence will very likely mean the effect sizes that are reported will be smaller. But they will reflect reality.

Expectations about how much impact interventions tend to have need to be massively recalibrated since more-rigorous research with larger sample sizes tends to find smaller effect sizes. If an educational intervention’s outcomes seem too good to be true, they probably are. (A recent working paper by Matt Kraft of Brown University provides a helpful overview of how we might think about effect sizes in education.)

There do not appear to be single “silver bullets” that will easily equalize and accelerate educational outcomes. The reality is that educational gains will come from a combination of many well-supported, evidence-based practices and communities of caring adults helping kids.

We hope educational policymakers and practitioners will start proactively looking for practices that meet these standards of evidence and work to widely implement them. By doing so, we can move steadily toward greater educational success for all students.

Follow the Education Week Opinion section on Twitter.

Sign up to get the latest Education Week Opinion in your email inbox.
A version of this article appeared in the October 30, 2019 edition of Education Week as Raising the Bar for Evidence in Education

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Equity, Care and Connection: New SEL Tools and Practices to Support Students and Adults
As school districts plan to welcome students back into buildings for the upcoming school year, this is the perfect time to take a hard look at both our practices and our systems to build a
Content provided by Panorama Education
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Classroom Technology Webinar
Here to Stay – Pandemic Lessons for EdTech in Future Development
What technology is needed in a post pandemic district? Learn how changes in education will impact development of new technologies.
Content provided by AWS
School & District Management Live Online Discussion A Seat at the Table: Strategies & Tips for Complex Decision-Making
Schools are working through the most disruptive period in the history of modern education, facing a pandemic, economic problems, social justice issues, and rapid technological change all at once. But even after the pandemic ends,

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management National Teachers' Union President: Schools Must Reopen 5 Days a Week This Fall
American Federation of Teachers President Randi Weingarten wants five days a week of in-person school next fall.
4 min read
Randi Weingarten, president of the American Federation of Teachers, talks during a news conference in front of the Richard R. Green High School of Teaching on Sept. 8, 2020.
Randi Weingarten, president of the American Federation of Teachers, talks during a news conference in front of the Richard R. Green High School of Teaching on Sept. 8, 2020.
Mark Lennihan/AP
School & District Management Principals and Stress: Strategies for Coping in Difficult Times
Running schools in the pandemic has strained leaders in unprecedented ways. Principals share their ideas for how to manage the stress.
6 min read
Illustration of calm woman working at desk
Getty
School & District Management Wanted: Superintendents to Lead Districts Through the End of a Pandemic
Former superintendents say there are signs when it's time to move on. Their replacements are more likely to be greenhorns, experts say.
4 min read
Los Angeles Unified School District Superintendent Austin Beutner speaks at a news conference at the school district headquarters in Los Angeles on March 13, 2020. Beutner will step down as superintendent after his contract ends in June, he announced Wednesday, April 21, 2021.
Austin Beutner, the superintendent of Los Angeles Unified, will step down after his contract ends in June.
Damian Dovarganes/AP
School & District Management Has COVID-19 Led to a Mass Exodus of Superintendents?
This year has been exhausting for superintendents. Some experts say they're seeing an unusually high number of resignations this spring.
5 min read
Chicago Public Schools Superintendent Janice K. Jackson, right, speaks on Feb. 11, 2021, during a news conference at the William H. Brown Elementary School in Chicago. In-person learning for students in pre-k and cluster programs began Thursday, since the district's agreement with the Chicago Teachers Union was reached.
Chicago Public Schools Superintendent Janice K. Jackson, right, announced earlier this week that she would depart the school system. Jackson, who assumed the superintendency in 2018, has worked for more than 20 years in CPS.
Shafkat Anowar