Opinion
Federal Opinion

Is ‘Evidence’ Enough?

By Harry Brighouse — September 25, 2006 7 min read
  • Save to favorites
  • Print

—Dave Cutler

BRIC ARCHIVE

There is much talk these days about “evidence based” policy in education and elsewhere. But empirical evidence alone, valuable as it is, can never be enough to determine what policy is best. One reason for this is that in the social sciences, even the best available evidence almost always leaves room for doubt and uncertainty. More fundamentally, however, the right thing to do always depends on values as well as facts, and values need systematic study just as much as facts.

Consider some of the ways in which evidence underdetermines policy. Although there is a movement for school choice, another for standards and accountability, and another for small schools, each of those movements unites around a broad idea, not a single specific reform. School choice comprises charter schools, vouchers, and open enrollment; “small schools” comprises small schools, schools within schools, and the Harry Potter-ish “smaller learning communities.” Each state has its own charter legislation; the Milwaukee, Cleveland, and Florida voucher systems are all quite different from one another. My guess is that there are as many different variants of smaller learning communities as there are schools adopting them.

But the success or failure of any reform depends on a myriad of contextual factors. Small-scale studies of particular instances of a reform idea can be sensitive to context, but finding out that a reform produces certain results in one place doesn’t tell us what results it will produce elsewhere. Large-scale studies of several variants of a reform may tell us whether to bet on it when it is introduced in a new context. But the results range over a wide variety of reform designs in a number of different contexts to which the study is not sensitive. Knowing that 10 variants of some reform have a particular consequence on average doesn’t tell you whether it will have that consequence in your circumstance.

In addition, depending on the level at which you are contemplating reform, you face the “scaling up” problem. Suppose you are a state superintendent, and the research tells you that pretty much wherever districts have invested heavily in professional development, it seems to have paid off. Should you do the same? What the microlevel studies do not usually tell you is the extent to which the payoff is the result of improved quality in the existing teachers, and the extent to which the investment has simply attracted and retained the better teachers in the regional labor market. To the extent that the former mechanism is at work, the strategy is likely to work for the state; to the extent that it is the latter, the prospective payoff at the scaled-up level is low.

So at best the would-be reformer is deciding under uncertainty. All these problems could be addressed by more and better (and more expensive) empirical studies; and I’m sure my colleagues in schools of education would welcome more of those.

But suppose that you had perfect information about the prospective effects of every reform option. Even this would not tell you what to choose, absent reflection on the values that should inform schooling. Suppose, for example, that a reform raises test scores for the whole population, but raises the higher test scores by more than it does the lower test scores; a rising tide that increases the gap. Is this better than an alternative that raises the lower scores but lowers the higher scores (an uneven tide that reduces the gap)? Or compare a reform that improves math scores across the board, but at the cost of schools’ de-emphasizing pedagogical goals in which success is less easy to measure, like responsible citizenship, or a lifelong love of literature. A nation of highly numerate but economically self-interested voters is not clearly preferable to a nation of good-willed and reasonable deliberators who have difficulty with calculus.

Despite the fact that ethical questions arise all the way down the educational structure, there isn’t a public school, school district, or state department of public instruction with a staff ethicist.

Skeptics might think these are “philosopher’s examples”—interesting in the abstract, but irrelevant to the real world, where there are no real trade-offs. I doubt this. A recent study of schooling in Amsterdam found that low-income Turkish and Moroccan students achieved better academic results when concentrated into the same schools than when integrated into schools with native Dutch students—a prima facie case of a trade-off between closing the achievement gap and improving social solidarity. Closer to home, consider Nathan Glazer’s comment in his review of Jonathan Kozol’s 2005 book The Shame of the Nation:

“To be sure, the case for both [racial] integration and equality of expenditure is powerful. But the chief obstacle to achieving these goals does not seem to be the indifference of whites and the nonpoor to the education of nonwhites and the poor. … Rather, other values, which are not simply shields for racism, stand in the way: the value of the neighborhood school; the value of local control of education, and, above all, the value of freedom from state imposition when it affects matters so personal as the future of one’s children.”

Glazer claims that in our world, when considering real reform ideas, these values come into conflict, and it is hard to believe that he’s wrong, even if you ultimately side with Kozol’s way of making the trade-off.

Surely policymakers just apply their own values to the trade-offs? Well, maybe they do, but doing so is rather like making empirical judgments on the basis of their own experience. These are values that really matter, and there are better and worse ways of adjudicating between them in the real world of conflict. Careful philosophical reflection is needed, and this is something that requires research. Indeed, there is a good deal of philosophical research on the liberty-equality trade-off, and my reading of it is that it points strongly in favor of Kozol’s way of deciding, if feasibility constraints allow.

So policymakers need to think carefully about the values in play when they approach reforms, and to take a particularly careful look at trade-offs between values. Philosophical research can help.

Now, a warning. Philosophical research does not determine how a policymaker should act, any more than empirical research does. It is almost useless in the absence of the relevant empirical research. Even in combination with the empirical research, it is not usually fully determinate, because a policymaker has to weigh all the relevant value considerations in the circumstances, and it is a rare piece of philosophical research that considers all things. A policymaker who slavishly took up a particular philosophical perspective would be as bad as one who considered only the empirical research that supported his case.

But I’d hazard a guess that fewer policymakers use philosophical research than use empirical research in making their decisions. The contrast with health policymakers is interesting. Health policymakers and medical practitioners are keenly aware not only that their choices involve making trade-offs between different values, but also that they are not able to pull a value off the shelf. They understand their obligation to scrutinize rigorously the ethical issues in question. Every hospital has a staff ethicist and an ethics board. The National Institutes of Health and the World Health Organization have several ethicists pursuing original research and consulting with decisionmakers.

Despite the fact that ethical questions arise all the way down the educational structure—from the federal government choosing whether to allocate budgets to children from low-income backgrounds or with disabilities, through school districts deciding whether to institutionalize acts of patriotic observance, to teachers deciding whether to inform parents of their suspicion that a child is experimenting with drugs—there isn’t a public school, public school district, or state department of public instruction with a staff ethicist.

Medical and health-policy ethics did not emerge as a discipline relevant to and informative for policy and practice simply because it was wished up by the profession.

Educational policymakers bear almost none of the blame for this state of affairs. Social-science academics view questions of value with suspicion; quantitative scholars try to avoid value judgments, while qualitative scholars treat value judgments as inevitable biases. Philosophers, by contrast, have by and large failed to involve themselves in the real world of education policy and research. They are trained to ignore questions of feasibility, which bedevil well-willed policymakers and constrain the extent to which important values can be realized. So their findings are rarely presented in ways that are readily interpreted and usable by social scientists and policymakers.

Medical and health-policy ethics did not emerge as a discipline relevant to and informative for policy and practice simply because it was wished up by the profession. Philosophers had consciously to learn about the medical world, and, while continuing to do philosophical work at an abstract level, simultaneously deliberate about the concrete problems that the profession faced, in a way that would be accessible to nonprofessional philosophers.

Policymakers need resources to help them think through the questions of value as rigorously as possible, and they need empirical research that enables them to choose the variants of the reforms in play that have the best prospects for realizing the most important values. These resources do not come out of thin air, but, as the case of medical and health-policy ethics suggests, through an ongoing dialogue involving all three parties.

A version of this article appeared in the September 27, 2006 edition of Education Week as Is ‘Evidence’ Enough?

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Webinar
The State of Assessment in K-12 Education
What is the impact of assessment on K-12 education? What does that mean for administrators, teachers and most importantly—students?
Content provided by Instructure
Jobs January 2022 Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Reading & Literacy Webinar
Proven Strategies to Improve Reading Scores
In this webinar, education and reading expert Stacy Hurst will provide a look at some of the biggest issues facing curriculum coordinators, administrators, and teachers working in reading education today. You will: Learn how schools
Content provided by Reading Horizons

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Miguel Cardona Came in as a Teacher Champion. Has COVID Muted His Message?
The education secretary is taking heat from some who say his advocacy is overshadowed by Biden's push to keep schools open.
11 min read
Secretary of Education Miguel Cardona talks to students at White Plains High School in White Plains, N.Y. on April 22, 2021.
U.S. Secretary of Education Miguel Cardona talks to students at White Plains High School in White Plains, N.Y., last April.
Mark Lennihan/AP
Federal Citing Educator and Parent Anxieties, Senators Press Biden Officials on Omicron Response
Lawmakers expressed concern about schools' lack of access to masks and coronavirus tests, as well as disruptions to in-person learning.
5 min read
Dr. Rochelle Walensky, Director of the Centers for Disease Control and Prevention, left, and Dr. Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases and chief medical adviser to the president, testify before a Senate Health, Education, Labor, and Pensions Committee hearing to examine the federal response to COVID-19 and new emerging variants, Tuesday, Jan. 11, 2022 on Capitol Hill in Washington.
Dr. Rochelle Walensky, director of the Centers for Disease Control and Prevention, left, and Dr. Anthony Fauci, chief medical adviser to the president, testify at a Senate hearing about the federal response to COVID-19.
Greg Nash/Pool via AP
Federal Miguel Cardona Should Help Schools Push Parents to Store Guns Safely, Lawmakers Say
More than 100 members of Congress say a recent shooting at a Michigan high school underscores the need for Education Department action.
3 min read
Three Oakland County Sheriff's deputies survey the grounds outside of the residence of parents of the Oxford High School shooter on Dec. 3, 2021, in Oxford, Mich.
Three Oakland County Sheriff's deputies survey the grounds outside of the Crumbley residence while seeking James and Jennifer Crumbley, parents of Oxford High School shooter Ethan Crumbley, on Dec. 3, 2021, in Oxford, Mich.
Jake May/The Flint Journal via AP
Federal In Reversal, Feds Seek to Revive DeVos-Era Questions About Sexual Misconduct by Educators
The Education Department's decision follows backlash from former education Secretary Betsy DeVos and other conservatives.
4 min read
Illustration of individual carrying binary data on his back to put back into the organized background of 1s and 0s.
iStock/Getty Images Plus