Without training in research methodology or access to a researcher’s data, it’s often difficult to tell how objective a study may be. But experts agree that educators and policymakers can find some clues about a study’s potential biases by asking themselves certain questions about the study’s design and the researcher’s motives. Here are examples of the kinds of questions that researcher say may be useful to keep in mind:
- Has the study been reviewed or critiqued by the investigator’s colleagues?
- Does the researcher take care to address all of the arguments that could refute his or her findings?
- Has the researcher made the study data available so that other researchers can check the work?
- Has the researcher used methods that are appropriate for answering the questions posed in the study? One would not draw conclusions on the overall effectiveness of a school reform program, for example, by tracking what happens in one school using the program over the course of one year.
- If the study uses a quasi-experimental design, are the treatment and comparison groups large enough so that the improvements measured are not easily explained by other circumstances in those classrooms or schools? Are the study samples skewed in some way?
- Does the researcher make claims in the news media for the study’s findings that cannot be supported by a closer look at the report itself?
- Does it seem as if the researcher is posing a loaded questionone for which he or she may already know the answer?
- Does the researcher discuss the limitations of the research in the report?
- Who is paying for the study, and who stands to benefit from a particular result?
