Opinion
Federal Opinion

Is ‘Evidence’ Enough?

By Harry Brighouse — September 25, 2006 7 min read
  • Save to favorites
  • Print

—Dave Cutler

BRIC ARCHIVE

There is much talk these days about “evidence based” policy in education and elsewhere. But empirical evidence alone, valuable as it is, can never be enough to determine what policy is best. One reason for this is that in the social sciences, even the best available evidence almost always leaves room for doubt and uncertainty. More fundamentally, however, the right thing to do always depends on values as well as facts, and values need systematic study just as much as facts.

Consider some of the ways in which evidence underdetermines policy. Although there is a movement for school choice, another for standards and accountability, and another for small schools, each of those movements unites around a broad idea, not a single specific reform. School choice comprises charter schools, vouchers, and open enrollment; “small schools” comprises small schools, schools within schools, and the Harry Potter-ish “smaller learning communities.” Each state has its own charter legislation; the Milwaukee, Cleveland, and Florida voucher systems are all quite different from one another. My guess is that there are as many different variants of smaller learning communities as there are schools adopting them.

But the success or failure of any reform depends on a myriad of contextual factors. Small-scale studies of particular instances of a reform idea can be sensitive to context, but finding out that a reform produces certain results in one place doesn’t tell us what results it will produce elsewhere. Large-scale studies of several variants of a reform may tell us whether to bet on it when it is introduced in a new context. But the results range over a wide variety of reform designs in a number of different contexts to which the study is not sensitive. Knowing that 10 variants of some reform have a particular consequence on average doesn’t tell you whether it will have that consequence in your circumstance.

In addition, depending on the level at which you are contemplating reform, you face the “scaling up” problem. Suppose you are a state superintendent, and the research tells you that pretty much wherever districts have invested heavily in professional development, it seems to have paid off. Should you do the same? What the microlevel studies do not usually tell you is the extent to which the payoff is the result of improved quality in the existing teachers, and the extent to which the investment has simply attracted and retained the better teachers in the regional labor market. To the extent that the former mechanism is at work, the strategy is likely to work for the state; to the extent that it is the latter, the prospective payoff at the scaled-up level is low.

So at best the would-be reformer is deciding under uncertainty. All these problems could be addressed by more and better (and more expensive) empirical studies; and I’m sure my colleagues in schools of education would welcome more of those.

But suppose that you had perfect information about the prospective effects of every reform option. Even this would not tell you what to choose, absent reflection on the values that should inform schooling. Suppose, for example, that a reform raises test scores for the whole population, but raises the higher test scores by more than it does the lower test scores; a rising tide that increases the gap. Is this better than an alternative that raises the lower scores but lowers the higher scores (an uneven tide that reduces the gap)? Or compare a reform that improves math scores across the board, but at the cost of schools’ de-emphasizing pedagogical goals in which success is less easy to measure, like responsible citizenship, or a lifelong love of literature. A nation of highly numerate but economically self-interested voters is not clearly preferable to a nation of good-willed and reasonable deliberators who have difficulty with calculus.

Despite the fact that ethical questions arise all the way down the educational structure, there isn’t a public school, school district, or state department of public instruction with a staff ethicist.

Skeptics might think these are “philosopher’s examples”—interesting in the abstract, but irrelevant to the real world, where there are no real trade-offs. I doubt this. A recent study of schooling in Amsterdam found that low-income Turkish and Moroccan students achieved better academic results when concentrated into the same schools than when integrated into schools with native Dutch students—a prima facie case of a trade-off between closing the achievement gap and improving social solidarity. Closer to home, consider Nathan Glazer’s comment in his review of Jonathan Kozol’s 2005 book The Shame of the Nation:

“To be sure, the case for both [racial] integration and equality of expenditure is powerful. But the chief obstacle to achieving these goals does not seem to be the indifference of whites and the nonpoor to the education of nonwhites and the poor. … Rather, other values, which are not simply shields for racism, stand in the way: the value of the neighborhood school; the value of local control of education, and, above all, the value of freedom from state imposition when it affects matters so personal as the future of one’s children.”

Glazer claims that in our world, when considering real reform ideas, these values come into conflict, and it is hard to believe that he’s wrong, even if you ultimately side with Kozol’s way of making the trade-off.

Surely policymakers just apply their own values to the trade-offs? Well, maybe they do, but doing so is rather like making empirical judgments on the basis of their own experience. These are values that really matter, and there are better and worse ways of adjudicating between them in the real world of conflict. Careful philosophical reflection is needed, and this is something that requires research. Indeed, there is a good deal of philosophical research on the liberty-equality trade-off, and my reading of it is that it points strongly in favor of Kozol’s way of deciding, if feasibility constraints allow.

So policymakers need to think carefully about the values in play when they approach reforms, and to take a particularly careful look at trade-offs between values. Philosophical research can help.

Now, a warning. Philosophical research does not determine how a policymaker should act, any more than empirical research does. It is almost useless in the absence of the relevant empirical research. Even in combination with the empirical research, it is not usually fully determinate, because a policymaker has to weigh all the relevant value considerations in the circumstances, and it is a rare piece of philosophical research that considers all things. A policymaker who slavishly took up a particular philosophical perspective would be as bad as one who considered only the empirical research that supported his case.

But I’d hazard a guess that fewer policymakers use philosophical research than use empirical research in making their decisions. The contrast with health policymakers is interesting. Health policymakers and medical practitioners are keenly aware not only that their choices involve making trade-offs between different values, but also that they are not able to pull a value off the shelf. They understand their obligation to scrutinize rigorously the ethical issues in question. Every hospital has a staff ethicist and an ethics board. The National Institutes of Health and the World Health Organization have several ethicists pursuing original research and consulting with decisionmakers.

Despite the fact that ethical questions arise all the way down the educational structure—from the federal government choosing whether to allocate budgets to children from low-income backgrounds or with disabilities, through school districts deciding whether to institutionalize acts of patriotic observance, to teachers deciding whether to inform parents of their suspicion that a child is experimenting with drugs—there isn’t a public school, public school district, or state department of public instruction with a staff ethicist.

Medical and health-policy ethics did not emerge as a discipline relevant to and informative for policy and practice simply because it was wished up by the profession.

Educational policymakers bear almost none of the blame for this state of affairs. Social-science academics view questions of value with suspicion; quantitative scholars try to avoid value judgments, while qualitative scholars treat value judgments as inevitable biases. Philosophers, by contrast, have by and large failed to involve themselves in the real world of education policy and research. They are trained to ignore questions of feasibility, which bedevil well-willed policymakers and constrain the extent to which important values can be realized. So their findings are rarely presented in ways that are readily interpreted and usable by social scientists and policymakers.

Medical and health-policy ethics did not emerge as a discipline relevant to and informative for policy and practice simply because it was wished up by the profession. Philosophers had consciously to learn about the medical world, and, while continuing to do philosophical work at an abstract level, simultaneously deliberate about the concrete problems that the profession faced, in a way that would be accessible to nonprofessional philosophers.

Policymakers need resources to help them think through the questions of value as rigorously as possible, and they need empirical research that enables them to choose the variants of the reforms in play that have the best prospects for realizing the most important values. These resources do not come out of thin air, but, as the case of medical and health-policy ethics suggests, through an ongoing dialogue involving all three parties.

A version of this article appeared in the September 27, 2006 edition of Education Week as Is ‘Evidence’ Enough?

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Artificial Intelligence Webinar
AI in Education: Empowering Educators to Tap into the Promise and Steer Clear of Peril
Explore the transformative potential of AI in education and learn how to harness its power to improve student outcomes.
Content provided by Panorama Education
English Learners Webinar Family and Community Engagement: Best Practices for English Learners
Strengthening the bond between schools and families is key to the success of English learners. Learn how to enhance family engagement and support student achievement.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Mathematics Webinar
How an Inquiry-Based Approach Transforms Math Learning
Transform math learning with an approach that empowers students to become active, engaged learners.
Content provided by MIND Education

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Opinion A Reminder to America: We Still Agree on More Than We Don’t
In the wake of the election, the education sector has important work ahead to bridge our political differences.
Katy Anthes
4 min read
Hands reach out to each other. Handshake.
<br/>Alona Horkova/iStock + Education Week
Federal Q&A Betsy DeVos' Advice for Trump's Next Education Secretary
DeVos is eager to see a second Trump administration finish what she helped start on school choice and rolling back the federal K-12 role.
6 min read
Education Secretary Betsy DeVos speaks during a briefing at the Department of Education building in Washington on July 8, 2020.
Education Secretary Betsy DeVos speaks during a briefing at the Department of Education building in Washington on July 8, 2020.
Manuel Balce Ceneta/AP
Federal Can Trump Really Dismantle the Department of Education?
Republicans have long threatened to axe the U.S. Department of Education, but doing so would be complicated.
6 min read
The U.S. Department of Education in Washington, D.C., pictured on Tuesday, August 23, 2022.
The U.S. Department of Education in Washington, D.C., pictured on Tuesday, August 23, 2022.
Alyssa Schukar for Education Week
Federal Trump's Push to Expand Choice, Nix the Ed. Dept. Takes on New Momentum
Trump’s decisive victory doused accelerant over his plans to dismantle the Education Department and expand school choice.
5 min read
Republican presidential nominee former President Donald Trump, left, with Republican vice presidential nominee Sen. JD Vance, R-Ohio, right, at an election night watch party at the Palm Beach Convention Center on Nov. 6, 2024, in West Palm Beach, Fla.
President-elect Donald Trump, left, with Vice President-elect JD Vance at an election night watch party at the Palm Beach Convention Center on Nov. 6, 2024, in West Palm Beach, Fla. Trump's education policy priorities of expanding school choice, cutting federal education spending, and abolishing the Education Department have taken on new energy with his decisive victory Tuesday in the presidential election.
Evan Vucci/AP