Brightening the line between educational research and policy-marketing.
As Congress prepares to reauthorize federal programs in educational research, there is a growing controversy within the field over whether advocacy has a place in such research. Beneath the surface are telltale signs of an especially disturbing new competitor in conceptions of the role of this research, one that jettisons the idea of educational research as a source of objective information for policymaking, and mangles it into a tool for marketing partisan policy preferences.
The principle that guides marketing is simple: Get people to buy your wares. Principles such as honesty, objectivity, demanding good evidence, entertaining counter-evidence, and so forth—ideals that regulate research of all kinds—have little or no role to play. Indeed, they just get in the way.
Think about how people use research to market cars. They search for some study to support their nameplate. It is to their advantage if they can find a study that has been conducted by a reputable and impartial organization. For example, several carmakers tout their high ratings on crash tests conducted by the National Highway Traffic Safety Administration. Of course, carmakers don’t mention it if their crash-worthy nameplate is expensive, ugly, unreliable, or a gas guzzler. They never say whether findings from other studies conflict with the one they’re citing, and they will find any supportive study they can, from a reputable source or not. Finally, when other possibilities are exhausted, they pay for their own studies.
I want to suggest that educational policies are being marketed in much the same way. Educational research is being used as a spruced-up form of testimonial: “ABC Foundation conducted a study of X in which researchers concluded that ...” Such testimonials trade on the prestige of research while eschewing the principles that make it worthy of the name.
Consider the principle of peer review. Chester E. Finn Jr. has remarked: “I don’t have much use for peer review in education research. ... By selecting the peers, you’re preordaining the outcome of the review.” (“Research: Researching the Researchers,” Feb. 20, 2002.) This is a startling statement in light of reports in these pages that it is Mr. Finn, along with the educational historian and former federal official Diane Ravitch, who review the reports produced by the Thomas B. Fordham Foundation in Washington, an organization Mr. Finn heads. He and Ms. Ravitch, Mr. Finn would have to say, “preordain the outcomes” of the reviews of Fordham Foundation studies.
If he hadn’t already, the former assistant U.S. secretary of education for research lays his cards fully on the table when he adds, in that Feb. 20, 2002, article: “We’re engaged in an argument. ... We’re not refereeing an argument.” He may as well have said, “We’re engaged in marketing our policies.” Mr. Finn seems perfectly willing to abandon the principles that regulate the evaluation of research and turn educational policymaking over to a market mechanism. But there is a tension, if not a deception, in his modus operandi. Part of the marketing strategy is to trade on the expectation that because the proposals have the backing of research (so-called), they are not preordained by his policy preferences. And that is the function served by introducing communications into policy debates that take the form, “A recent study by the Fordham Foundation concluded ... “
Educational research is being used as a spruced-up form of testimonial. Such testimonials trade on the prestige of research while eschewing the principles that make it worthy of the name.
Joseph G. Lehman, the executive vice president of the Michigan-based Mackinac Center for Public Policy, is more evasive than Mr. Finn. He largely dodges the question of whether peer review is a means by which to help ensure the soundness of research, giving instead his reasons for eschewing rigorous peer review. Among these are that policymakers don’t read academic journals, and that while the Mackinac Center might be able to make “marginal improvements” in its review process, it could do so only at the cost of a loss of timeliness and relevance (read: influence) vis-à-vis policymakers. It never occurs to Mr. Lehman to think about how the process of policymaking might be improved by, for example, undertaking efforts to make research of the kind available in research journals—at least its conclusions, recommendations, and caveats—accessible to policymakers and the public. But it makes sense that Mr. Lehman wouldn’t point in this direction. That would be endorsing what Mr. Finn disdains as “refereeing,” an odd thing for an organization that is unabashedly committed to promoting free-market policies.
Like Mr. Finn, Mr. Lehman and the Mackinac Center cannot afford to forgo the impression that their research helps referee among contested views, because trading on that impression is part of the marketing strategy. Thus, when the Education Policy Studies Laboratory at Arizona State University published a critique of the research carried out by the Mackinac Center, the response consisted of some hand-waving regarding the lab’s failure to uncover any “outright errors” and its failure to have its own report rigorously reviewed. The Mackinac Center went on to intimate that the lab’s study was biased because it was funded by a Michigan teachers’ union. Although it raises a flag that the teachers’ union is a known “foe” of the Mackinac Center, so does the source of funds for Mackinac Center reports critical of unions. Establishing bias in either case requires evidence and argument that speak to the charge, but the Mackinac Center opted for the rhetorical ploy of “poisoning the well.”
Those who would reject this market free-for-all in educational research are plagued by the almost universally shared belief among educational researchers nowadays that educational research can never be value-free. For how can there be any objectivity where values are concerned? And where there is no such objectivity, how does one distinguish selling educational policy with the help of the testimony of “studies” from selling Pepsi with the help of the testimony of rock stars?
In one shape or another, questions about the objectivity of values have stimulated a philosophical debate spanning thousands of years, probably beginning with the wrangling between Socrates and the sophist Thrasymachus. So we shouldn’t expect an easy, knock-down answer to the question of how educational research can be laden with values and yet measure up to standards of objectivity. Still, there are good reasons to believe it can (and should).
It strains credulity to suggest that no objective standards exist that enable educational researchers to detect poor or biased research.
The idea that educational research can never be “value free” is often used interchangeably with the idea that it can never be “value neutral.” This is a mistake in my view. Although the difference is perhaps a fine one, it is important nonetheless. Whereas educational research can never be free of the commitment to some value-laden framework, it can, and often does, address questions that are neutral among value-laden frameworks.
For example, consider the question of whether school choice exacerbates stratification in terms of race and income. Determining the answer to this question has (or should have) nothing to do with whether one is an advocate or a critic of school choice. Indeed, it is just the sort of question that may be used to test the adequacy of competing views. Of course, people committed to one side or the other of the debate will be inclined to more tenaciously defend their view and may even try to shift ground—for example, by advocating that more or less emphasis, as the case may be, be placed on the importance of stratification in evaluating school choice policies. But all that means is that controversies about educational policies are complex and hard to resolve.
It strains credulity to suggest that no objective standards exist that enable educational researchers to detect poor or biased research—research, for example, that employs shoddy methodology; that suppresses and spins evidence; or that avoids entertaining relevant questions because the answers to them would likely damage its cause. Consider research on school choice once again. Studies that ignore the factor of stratification are biased in one direction; those that ignore the factor of parental satisfaction are biased in the other. And these are precisely the sorts of criticisms that the two sides advance against one another, whatever their rhetoric about the impossibility of objective standards for evaluating research.
An important reason to worry about the possibility of objectivity in the conduct and evaluation of educational research is its link to the role educational research plays—or ought to—in democratic decisionmaking. Conceptions of educational research are unavoidably linked to some conception of democratic decisionmaking by virtue of unavoidably assuming some stance toward “stakeholders": who qualifies for participation, what their roles should be vis-à-vis researchers, and what their needs for and rights to information might be. Some conceptions of decisionmaking aspire to the ideal of objectivity in dealing with stakeholders. Some do not.
The market conception is an instance of the latter. It embraces a no-holds-barred contest whereby interest groups compete to win assent to their educational policy preferences. As a consequence, it reduces democratic decisionmaking to a sham. It is highly unlikely that any but the rich and powerful can get their interests represented in decisionmaking processes carried out according to market rules.
The role of educational researchers is to use their scholarly knowledge and methodological expertise to ensure that the issues relevant to the topic of research are addressed, rigorously and evenhandedly.
An alternative conception of decisionmaking that does aspire to objectivity is the democratic one, which actively seeks to blunt imbalances of power in negotiating educational policy. It aspires to objectivity in the sense of seeking to ensure that all stakeholder views are represented in a substantive way, and are taken into account in deliberating about educational policy. The role of educational researchers is to use their scholarly knowledge and methodological expertise to ensure that the issues relevant to the topic of research are addressed, rigorously and evenhandedly.
We live in a complex society, in which the expert tools of social research are needed to effectively investigate complex educational policies. The democratic conception places a responsibility on educational researchers to do their best to inform educational policy debates by providing policymakers and the public with access to the best research available. Those who market their partisan preferences under the guise of informing public deliberation sully the research enterprise and violate the public trust.
Kenneth R. Howe is a professor of education at the University of Colorado at Boulder and the director of the newly formed Education and the Public Interest Center.
