Innovative Reforms Require Innovative Scorekeeping

By Lisbeth B. Schorr

President Barack Obama has made clear that we must systematically identify “what works,” both for budgetary reasons and to ensure that public money supports effective social programs and policies. The president and his budget chief recognize how tricky it is to make that determination. In June, Office of Management and Budget Director Peter Orszag released a statement describing how the administration will make sure that spending decisions are based not only on good intentions but also on strong evidence.

Serious social reformers today agree that rigorous efforts to determine “what works” are essential. But, depending on what the administration considers “strong evidence,” these efforts risk sabotaging or marginalizing some of the most innovative attempts to solve intractable social problems. I worry that, in defining what constitutes “the best available evidence” of effectiveness, the OMB and federal agencies will follow the constricted approach of the Coalition for Evidence-Based Policy and the U.S. Department of Education’s What Works Clearinghouse. These and similar organizations claim scientific rigor by insisting that public and philanthropic support go only to programs shown to be evidence-based through experimental evaluation methods , preferably involving random assignment of participants to experimental and control groups. The implication is that this methodology can determine definitively and objectively, uncontaminated by human judgment, whether any intervention—be it a pill, a model program, or an ambitious institutional change—produces a different outcome than would otherwise occur.

Unfortunately, no single, circumscribed program can turn things around in an entire community or for a whole population. Nor can complex social programs and policies be tested like new drugs. The interventions that turn around inner-city schools, strengthen families, and rebuild neighborhoods are not stable chemicals manufactured and administered in standardized doses. They are sprawling efforts with multiple components, some of which may be proven experimentally, but many that can’t be because they require midcourse corrections and adaptations...