Opinion Blog

Rick Hess Straight Up

Education policy maven Rick Hess of the American Enterprise Institute think tank offers straight talk on matters of policy, politics, research, and reform. Read more from this blog.

Education Opinion

How, Exactly, Is Research Supposed to Improve Education?

By Guest Blogger — September 02, 2020 6 min read
  • Save to favorites
  • Print

This week Dylan Wiliam, eclectic Wales native and emeritus professor at University College London, takes over the blog. Dylan began his career as a math teacher in London (having followed his “jazz-folk” band to the capital city) before eventually stumbling into academe. His books include Creating the Schools Our Children Need and Leadership for Teacher Learning. Across a varied career, he has taught in urban public schools, directed a large-scale testing program, spent three years as senior research director at the Educational Testing Service (ETS), and served a number of roles in university administration, including dean of a school of education at King’s College London. Dylan, whose advice usually costs a pretty penny, will spend the week offering pro bono thoughts and tips to educators struggling to get school going this fall.

— Rick

As educators, schools and districts work to overcome the damage to students’ education caused by the coronavirus pandemic, it seems obvious that our efforts should be guided by the best evidence we can find about what is likely to be most effective.

The good news is that, over the last 20 or so years, there have been substantial improvements in the way that research findings are summarized and made available to practitioners and policymakers. Increasingly, educational researchers have stepped out of their “ivory towers” and tackled issues of immediate and direct relevance. And, just as importantly, they have taken seriously the task of communicating their research findings to those who might actually use them in real settings (see, for example, here and here).

The bad news is that producing meaningful syntheses of research turns out to be much more difficult in education than it is in, say, medicine. Too little attention is given to figuring out how even well-established research findings might be implemented in real schools and classrooms.

Let’s start with research synthesis. For many years, research reviews typically took a “narrative” approach. Researchers read the relevant studies and then figured out the best story to tell. Some reviewers were more systematic, tallying the number of positive and negative results in a particular field, but such an approach treated all experiments as if they were the same size and had the same size of impact. In 1976, Gene V. Glass proposed a technique that he called “meta-analysis” by which the results of different studies could be expressed on a common metric, called “effect size,” and thus compared more meaningfully. This approach is now the standard approach to research synthesis in the health sciences.

Unfortunately, as I explain here, meta-analysis is much harder to do well in education for a variety of reasons. Combining different reports on “cooperative learning” might group together studies with very different approaches to cooperative learning, studies with younger students tend to produce larger effect sizes, and different ways of assessing student achievement can produce very different results for the same experiment. Also, since it is much easier to get studies published if the results are dramatic, the studies that are published tend to be the ones where everything went just right, so the published studies tend to overstate the effects in other settings.

These problems are compounded when the results of different meta-analyses are combined in a process sometimes called “meta-meta-analysis.” The number of assumptions made in these analyses make it impossible to determine what is really going on. The whole thing is reminiscent of the old joke about someone who, after a speed-reading course, said, “I was able to go through War and Peace in 20 minutes. It’s about Russia.”

Even when studies are conducted and reported carefully, it is not at all obvious that the results would be relevant across different contexts. While it may seem obvious that studies conducted on undergraduate students are unlikely to provide meaningful insights about how to teach kindergarten, or that those conducted in urban settings may not generalize to rural settings, that intuition can easily get lost when trying to digest the large amounts of information presented in a meta-analysis. It is also important to note that even when we have a well-designed randomized control trial, all we know about is the differences between the control and experimental groups. The schools and districts that chose to participate in the experiment may be different from those that did not.

Class-size-reduction studies are a case in point. Any class-size-reduction program requires additional teachers, so the quality of those teachers is a crucial determinant of the success of the program. Probably the best known such study—the Tennessee STAR study—required recruiting an extra 50 teachers, and it is reasonable to suppose that these were as good as those already employed. However, when such a program requires hundreds, or thousands of extra teachers, it is not at all clear that the additional teachers employed will be as good as those already working in the schools. Worse, the quality of available teachers is likely to vary from district to district. A class-size-reduction program may increase student achievement in one district, yet reduce it in another, due to the difficulty of recruiting good teachers. Similar arguments apply to multitiered systems of support. More intensive instruction in smaller groups is likely to be effective when those teaching the smaller groups are as effective as those teaching the class from which the students were withdrawn. But if those teaching the smaller groups are less effective, then a multitiered system may actually reduce student achievement.

The important point here is that those “on the ground"—the administrators in that district—will know far more about teacher recruitment than those in a state department of education. They have to look at whether the research solves a problem that the district has, how much extra student achievement will be generated, how much it will cost (in money and in educator time), and whether it is possible to implement the reform in their own setting. Research is essential in helping districts avoid things that are unlikely to benefit students, like catering to students’ preferred learning styles, and can identify some “best bets” for schools and districts, but research can never provide step-by-step instructions on how to improve student outcomes.

To simplify somewhat, everything works somewhere, and nothing works everywhere. The important question is, “Under what circumstances might this work” and that is something that those “on the ground” are best placed to determine.

Educators cannot afford to ignore research evidence—there are just too many “blind alleys” that look attractive but are unlikely to help students—but they have to choose judiciously. Some interventions may have small effects, but if they do not take up too much time, they can be highly cost-effective. Others may have larger effects but will take time and energy to implement, and, crucially, what works best for one district may not work for the next district. It is imperative, now in the midst of unprecedented educational challenges more than ever, that district leaders become “critical consumers” of research.

— Dylan

The opinions expressed in Rick Hess Straight Up are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Equity & Diversity Webinar
Evaluating Equity to Drive District-Wide Action this School Year
Educational leaders are charged with ensuring all students receive equitable access to a high-quality education. Yet equity is more than an action. It is a lens through which we continuously review instructional practices and student
Content provided by BetterLesson

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Schools Get the Brunt of Latest COVID Wave in South Carolina
In the past few weeks, South Carolina has set records for COVID-19 hospitalizations and new cases have approached peak levels of last winter.
4 min read
Two Camden Elementary School students in masks listen as South Carolina Gov. Henry McMaster talks about steps the school is taking to fight COVID-19, Wednesday, Sept. 15, 2021, in Camden, S.C. McMaster has adamantly and repeatedly come out against requiring masks in schools even as the average number of daily COVID-19 cases in the state has risen since early June. (AP Photo/Jeffrey Collins)
Education More States Are Requiring Schools to Teach Native American History and Culture
Advocates say their efforts have gained some momentum with the nation’s reckoning over racial injustice since the killing of George Floyd.
3 min read
A dancer participates in an intertribal dance at Schemitzun on the Mashantucket Pequot Reservation in Mashantucket, Conn., Saturday, Aug. 28, 2021. Connecticut and a handful of other states have recently decided to mandate students be taught about Native American culture and history. (AP Photo/Jessica Hill)
Education Judge's Temporary Order Allows Iowa Schools to Mandate Masks
A federal judge ordered the state to immediately halt enforcement of a law that prevents school boards from ordering masks to be worn.
4 min read
Iowa Gov. Kim Reynolds speaks to reporters following a news conference, Thursday, Aug. 19, 2021, in West Des Moines, Iowa. Reynolds lashed out at President Joe Biden Thursday after he ordered his education secretary to explore possible legal action against states that have blocked school mask mandates and other public health measures meant to protect students against COVID-19. Reynolds, a Republican, has signed a bill into law that prohibits school officials from requiring masks, raising concerns as delta variant virus cases climb across the state and schools resume classes soon. (AP Photo/Charlie Neibergall)
Education Hurricane Ida Deals New Blow to Louisiana Schools Struggling to Reopen
The opening of the school year offered teachers a chance to fully assess the pandemic's effects, only to have students forced out again.
8 min read
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021. Louisiana students, who were back in class after a year and a half of COVID-19 disruptions kept many of them at home, are now missing school again after Hurricane Ida. A quarter-million public school students statewide have no school to report to, though top educators are promising a return is, at most, weeks away, not months.
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021.
John Locher/AP