Federal Opinion

Is ‘Evidence’ Enough?

By Harry Brighouse — September 25, 2006 7 min read
  • Save to favorites
  • Print

—Dave Cutler


There is much talk these days about “evidence based” policy in education and elsewhere. But empirical evidence alone, valuable as it is, can never be enough to determine what policy is best. One reason for this is that in the social sciences, even the best available evidence almost always leaves room for doubt and uncertainty. More fundamentally, however, the right thing to do always depends on values as well as facts, and values need systematic study just as much as facts.

Consider some of the ways in which evidence underdetermines policy. Although there is a movement for school choice, another for standards and accountability, and another for small schools, each of those movements unites around a broad idea, not a single specific reform. School choice comprises charter schools, vouchers, and open enrollment; “small schools” comprises small schools, schools within schools, and the Harry Potter-ish “smaller learning communities.” Each state has its own charter legislation; the Milwaukee, Cleveland, and Florida voucher systems are all quite different from one another. My guess is that there are as many different variants of smaller learning communities as there are schools adopting them.

But the success or failure of any reform depends on a myriad of contextual factors. Small-scale studies of particular instances of a reform idea can be sensitive to context, but finding out that a reform produces certain results in one place doesn’t tell us what results it will produce elsewhere. Large-scale studies of several variants of a reform may tell us whether to bet on it when it is introduced in a new context. But the results range over a wide variety of reform designs in a number of different contexts to which the study is not sensitive. Knowing that 10 variants of some reform have a particular consequence on average doesn’t tell you whether it will have that consequence in your circumstance.

In addition, depending on the level at which you are contemplating reform, you face the “scaling up” problem. Suppose you are a state superintendent, and the research tells you that pretty much wherever districts have invested heavily in professional development, it seems to have paid off. Should you do the same? What the microlevel studies do not usually tell you is the extent to which the payoff is the result of improved quality in the existing teachers, and the extent to which the investment has simply attracted and retained the better teachers in the regional labor market. To the extent that the former mechanism is at work, the strategy is likely to work for the state; to the extent that it is the latter, the prospective payoff at the scaled-up level is low.

So at best the would-be reformer is deciding under uncertainty. All these problems could be addressed by more and better (and more expensive) empirical studies; and I’m sure my colleagues in schools of education would welcome more of those.

But suppose that you had perfect information about the prospective effects of every reform option. Even this would not tell you what to choose, absent reflection on the values that should inform schooling. Suppose, for example, that a reform raises test scores for the whole population, but raises the higher test scores by more than it does the lower test scores; a rising tide that increases the gap. Is this better than an alternative that raises the lower scores but lowers the higher scores (an uneven tide that reduces the gap)? Or compare a reform that improves math scores across the board, but at the cost of schools’ de-emphasizing pedagogical goals in which success is less easy to measure, like responsible citizenship, or a lifelong love of literature. A nation of highly numerate but economically self-interested voters is not clearly preferable to a nation of good-willed and reasonable deliberators who have difficulty with calculus.

Despite the fact that ethical questions arise all the way down the educational structure, there isn’t a public school, school district, or state department of public instruction with a staff ethicist.

Skeptics might think these are “philosopher’s examples”—interesting in the abstract, but irrelevant to the real world, where there are no real trade-offs. I doubt this. A recent study of schooling in Amsterdam found that low-income Turkish and Moroccan students achieved better academic results when concentrated into the same schools than when integrated into schools with native Dutch students—a prima facie case of a trade-off between closing the achievement gap and improving social solidarity. Closer to home, consider Nathan Glazer’s comment in his review of Jonathan Kozol’s 2005 book The Shame of the Nation:

“To be sure, the case for both [racial] integration and equality of expenditure is powerful. But the chief obstacle to achieving these goals does not seem to be the indifference of whites and the nonpoor to the education of nonwhites and the poor. … Rather, other values, which are not simply shields for racism, stand in the way: the value of the neighborhood school; the value of local control of education, and, above all, the value of freedom from state imposition when it affects matters so personal as the future of one’s children.”

Glazer claims that in our world, when considering real reform ideas, these values come into conflict, and it is hard to believe that he’s wrong, even if you ultimately side with Kozol’s way of making the trade-off.

Surely policymakers just apply their own values to the trade-offs? Well, maybe they do, but doing so is rather like making empirical judgments on the basis of their own experience. These are values that really matter, and there are better and worse ways of adjudicating between them in the real world of conflict. Careful philosophical reflection is needed, and this is something that requires research. Indeed, there is a good deal of philosophical research on the liberty-equality trade-off, and my reading of it is that it points strongly in favor of Kozol’s way of deciding, if feasibility constraints allow.

So policymakers need to think carefully about the values in play when they approach reforms, and to take a particularly careful look at trade-offs between values. Philosophical research can help.

Now, a warning. Philosophical research does not determine how a policymaker should act, any more than empirical research does. It is almost useless in the absence of the relevant empirical research. Even in combination with the empirical research, it is not usually fully determinate, because a policymaker has to weigh all the relevant value considerations in the circumstances, and it is a rare piece of philosophical research that considers all things. A policymaker who slavishly took up a particular philosophical perspective would be as bad as one who considered only the empirical research that supported his case.

But I’d hazard a guess that fewer policymakers use philosophical research than use empirical research in making their decisions. The contrast with health policymakers is interesting. Health policymakers and medical practitioners are keenly aware not only that their choices involve making trade-offs between different values, but also that they are not able to pull a value off the shelf. They understand their obligation to scrutinize rigorously the ethical issues in question. Every hospital has a staff ethicist and an ethics board. The National Institutes of Health and the World Health Organization have several ethicists pursuing original research and consulting with decisionmakers.

Despite the fact that ethical questions arise all the way down the educational structure—from the federal government choosing whether to allocate budgets to children from low-income backgrounds or with disabilities, through school districts deciding whether to institutionalize acts of patriotic observance, to teachers deciding whether to inform parents of their suspicion that a child is experimenting with drugs—there isn’t a public school, public school district, or state department of public instruction with a staff ethicist.

Medical and health-policy ethics did not emerge as a discipline relevant to and informative for policy and practice simply because it was wished up by the profession.

Educational policymakers bear almost none of the blame for this state of affairs. Social-science academics view questions of value with suspicion; quantitative scholars try to avoid value judgments, while qualitative scholars treat value judgments as inevitable biases. Philosophers, by contrast, have by and large failed to involve themselves in the real world of education policy and research. They are trained to ignore questions of feasibility, which bedevil well-willed policymakers and constrain the extent to which important values can be realized. So their findings are rarely presented in ways that are readily interpreted and usable by social scientists and policymakers.

Medical and health-policy ethics did not emerge as a discipline relevant to and informative for policy and practice simply because it was wished up by the profession. Philosophers had consciously to learn about the medical world, and, while continuing to do philosophical work at an abstract level, simultaneously deliberate about the concrete problems that the profession faced, in a way that would be accessible to nonprofessional philosophers.

Policymakers need resources to help them think through the questions of value as rigorously as possible, and they need empirical research that enables them to choose the variants of the reforms in play that have the best prospects for realizing the most important values. These resources do not come out of thin air, but, as the case of medical and health-policy ethics suggests, through an ongoing dialogue involving all three parties.

A version of this article appeared in the September 27, 2006 edition of Education Week as Is ‘Evidence’ Enough?


Student Well-Being K-12 Essentials Forum Boosting Student and Staff Mental Health: What Schools Can Do
Join this free virtual event based on recent reporting on student and staff mental health challenges and how schools have responded.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
Practical Methods for Integrating Computer Science into Core Curriculum
Dive into insights on integrating computer science into core curricula with expert tips and practical strategies to empower students at every grade level.
Content provided by Learning.com

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Will the Government Actually Shut Down This Time? What Educators Should Know
The federal government is once again on the verge of shutting down. Here's why educators should care, but shouldn't necessarily worry.
1 min read
Photo illustration of Capitol building and closed sign.
Federal Biden Admin. Warns Schools to Protect Students From Antisemitism, Islamophobia
The U.S. Department of Education released a "Dear Colleague" letter reminding schools of their obligation to address discrimination.
3 min read
Education Secretary Miguel Cardona speaks during an interview with The Associated Press in his office at the Department of Education on Sept. 20, 2023 in Washington.
Education Secretary Miguel Cardona speaks during an interview in his office at the U.S. Department of Education on Sept. 20, 2023 in Washington.
Mark Schiefelbein/AP
Federal What Educators Should Know About Mike Johnson, New Speaker of the House
Johnson has supported restructuring federal education funding, as well as socially conservative policies that have become GOP priorities.
4 min read
House Speaker-elect Rep. Mike Johnson, R-La., addresses members of Congress at the Capitol in Washington on Oct. 25, 2023. Republicans eagerly elected Johnson as House speaker on Wednesday, elevating a deeply conservative but lesser-known leader to the seat of U.S. power and ending for now the political chaos in their majority.
House Speaker Mike Johnson, R-La., addresses members of Congress at the Capitol in Washington on Oct. 25, 2023. Johnson has a supported a number of conservative Republican education priorities in his time in Congress.
J. Scott Applewhite/AP
Federal America's Children Don't Have a Federal Right to Education. Will That Ever Change?
An education scholar is launching a new research and advocacy institute to make the case for a federal right to education.
6 min read
Kimberly Robinson speaks at the kickoff event for the new Education Rights Institute at the University of Virginia School of Law in Charlottesville, Va., on Oct. 16, 2023.
Kimberly Robinson speaks at the kickoff event for the new Education Rights Institute at the University of Virginia School of Law in Charlottesville, Va., on Oct. 16, 2023.
Julia Davis, University of Virginia School of Law