School & District Management Opinion

Education Research Comes Up Short

By Marc Tucker — August 07, 2014 6 min read
  • Save to favorites
  • Print

NCEE has been benchmarking the education systems of the top performing systems worldwide since 1989. This has prompted some reflections on the methods and substantive focus of education research, especially education research in the United States, which I will share in this and the next few blogs.

The achievements of the American education research establishment, with respect to the sophistication of method and the power of the findings, are very impressive. In every corner of the globe where we do our own research, we find people directing the most successful education reforms in the world talking about their debt to American researchers.

Yet something is amiss here. The American educational research establishment is by far the largest in the world. Yet the performance of the American education system is mediocre, and falling fast, relative to the achievements of the top performers. The whole point of public support for education research is the assumption that it is a very powerful driver, some would argue the most powerful driver, of improvement in education outcomes. What’s wrong? Is there less resistance to the findings of American education researchers outside the United States than within our own country? There is no evidence of that. Are the findings from our researchers more applicable in foreign than domestic settings? That would be bizarre.

The answer, I think, lies in the difference between the way the American education system is governed and the way the education systems in the top performing countries are governed, a subject about which I have written elsewhere. In the top performers, ministries of education at the national level and, in some countries, the state level, have virtually complete authority with respect to education policy. Not so in the United States, where the constitution greatly circumscribes the authority of the U.S. Department of Education, states’ departments of education are typically very weak, there are many other entities at the state level vested with responsibilities that, in another country, would be vested in the ministry, and local boards of education have far more authority, and often, much more staff, than their opposite numbers in the top performing countries. This is not decentralization. It is more accurately described as dispersion of authority.

The result is an education system that does not function as a system. In a properly functioning system, each of the parts and pieces is designed to work in harmony with all the other pieces. When one part adapts to changes in the environment, the related parts adapt accordingly. That is very much what we see in the top performing countries. But that is not what we see in the United States, where policies are rarely designed to support other parts and pieces and, in some cases are deliberately designed to frustrate the faithful execution of other policies, designed by people who have different policy objectives and embrace very different solutions to the issues of the day. In our system, elements of policy on the qualifications of teachers are often made by people who are at best unconnected to the people who set the standards for program of schools of education and both are different from the people who have the authority to waive the requirement for any standards for teachers and all of these are unrelated to the people who set teachers’ compensation and still others decide on most of the conditions of work. This set of facts makes it impossible to have a coherent human resources system in our schools. Each actor is at the mercy of all the others. In the top performing countries, examinations are based on a curriculum set by the state, but, in our country, the state might set the exams, but cannot do so based on the curriculum, because that is determined by the district, the school or the individual teacher. Curricula are set by people with very different values, interests and tends therefore to be a hodgepodge to which any given assessment may or may not be related. As a result, instructional systems in the United States typically lack the coherence found in the top performing countries.

The American approach to determining, through research, what works, is to identify some highly bounded problem, famously, for example, the teaching of reading, and look at contending programs, or in the medical terminology on which the research methodology is based, “treatments” that purport to address this challenge, and create, either physically or by statistical manipulation, a situation in which all variables other than the treatment are neutralized, so that the independent effect of the treatment can be measured.

Again and again, the effect sizes measured by these techniques are disappointingly small, but the search goes on. What is crucially important to grasp here is that, for forty years, the net change in performance of the American system is nearly zero, which no one has explained, and most researchers simply ignore. From my point of view, the explanation is obvious. The system itself is dysfunctional, for the reasons I have just described. Because all researched solutions live within this dysfunctional environment, one could expect little change if the system itself is not changed. Let me be a little clearer about what I mean by “dysfunctional.” Because policies are designed without reference to each other and all too often to counter one another by people who disagree with one another, it is hardly surprising that for any given actor, the incentives to do things that actually benefit students can at any given time be positive, neutral, negative or even perverse. In that environment the positive effects associated with one policy or practices are as likely as not to be cancelled out by the negative effects of another. I am speaking here, for example, about policies and practices having to do with curriculum, standards, assessment, teacher quality or school organization and management. The programs that would show the best results would not be the programs with the highest potential for success, but the programs that are best adapted to the demands of this dysfunctional system or those that are based on the most detailed and prescriptive designs, constructed that way in an effort to overcome at least some of the chaos.

And that is exactly what we see, everywhere. The most prescriptive programs and those that challenge the status quo the least are indeed the ones researchers find most effective.

But this is not the worst of it. If my colleagues and I are right, and the design of the system itself is the reason American schools are steadily falling behind the top performers, the biggest problem with our research effort is the lack of attention that our researchers pay to the design of the system itself. We are focusing with great sophistication on the constituent elements of the system, but hardly at all on what makes for effective systems. Our research has become a constant search for suboptimal solutions. When we start researching what it takes to create high performance systems, we will find that many programs now deemed to “work” will be found much less effective than policies that are not about programs but which, when joined with others, create systems in which students routinely perform at high levels.

I will return to this point in my next blog. For now, let me leave you with the thought that the real failure here is a failure of theory, not of methodology. American researchers, as I see it, have not really come to grips with the implications of the precipitous fall of the United States in the global education rankings. And, as a result, they have not really grasped the nettle, the importance of the design of education systems. They are focused on the trees, not the forest. They do not yet have a bundle of plausible hunches to explain the superiority of other nation’s systems on which theories could be developed and then tested. Until that happens, our very sophisticated research establishment will not be able to help us as much as it could. When we get some decent theories, we will see whether we need new methods to deal with them.

Follow NCEE @CtrEdEcon.

Related Tags:

The opinions expressed in Top Performers are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
What’s Next for Teaching and Learning? Key Trends for the New School Year
The past 18 months changed the face of education forever, leaving teachers, students, and families to adapt to unprecedented challenges in teaching and learning. As we enter the third school year affected by the pandemic—and
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
How Data and Digital Curriculum Can Drive Personalized Instruction
As we return from an abnormal year, it’s an educator’s top priority to make sure the lessons learned under adversity positively impact students during the new school year. Digital curriculum has emerged from the pandemic
Content provided by Kiddom
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management 10 Ways to Tackle Education's Urgent Challenges
As the school year gets underway, we ask hard questions about education’s biggest challenges and offer some solutions.
2 min read
Conceptual Image of schools preparing for the pandemic
Pep Montserrat for Education Week
School & District Management Reported Essay Principals Need Social-Emotional Support, Too
By overlooking the well-being of their school leaders, districts could limit how much their schools can flourish.
7 min read
Conceptual Illustration
Pep Montserrat for Education Week
School & District Management From Our Research Center Educator Stress, Anti-Racism, and Pandemic Response: How You're Feeling
A new nationally representative survey offers key takeaways from teachers, principals, and district leaders.
EdWeek Research Center
1 min read
2021 BI COVER no text DATA crop
Pep Montserrat for Education Week
School & District Management Download 8 Tips for Building a Digital Learning Plan That Conquers Chaos
Craft flexible strategies, encourage experimentation with new instructional models, and regularly solicit feedback.
1 min read
onsr edtech tips