Clearinghouse Responds to a Critic
To the Editor:
We write in response to the letters by Michael Pressley regarding the What Works Clearinghouse (“Education’s Clearinghouse,” Letters, July 28, 2004; “‘What Works’ Divergence,” Letters, Sept. 1, 2004). They raise important issues regarding our recently released reports and overall operation, which we address below.
Mr. Pressley notes that only a limited number of studies have passed the middle-school-math-curricula review. While true, that fact will not surprise many math experts, given the small research base in this area. Further, as part of our monthly release strategy, more WWC reports in this area are being prepared even as we write.
Mr. Pressley’s concerns regarding the information on studies identified as “Does not pass screen” are well-taken. In keeping with our commitment to transparency and continuous improvement, a fuller explanation of this process will be available soon at www.whatworks.ed.gov.
Another concern he expressed regards the use of peer review in the WWC process. We differ from previous efforts at research review and reporting, which often relied on expert panels to make judgments. Instead, the WWC has built an ongoing research-review enterprise that relies on two key elements: judgments guided by a set of pre-established, rigorous decision rules developed with the help of external experts, and extensive quality control to make sure the rules are applied in a consistent manner.
Further, the WWC’s quality-control process does include peer review at several critical points. For each topic, the WWC recruits external, nationally recognized senior content advisers, or scas, and methodology consultants, or mcs. All our reports are reviewed by the original study author (first author), the WWC’s technical advisory group, and the topic SCAS and MCS. The documents that guide the WWC rating process undergo peer review. The upcoming WWC “intervention reports” will be peer-reviewed.
Perhaps Mr. Pressley’s greatest concern is the inclusion of a study in our review of peer-assisted learning, or pal, in which he participated as a secondary author (Lysynchuk, Pressley, & Vye, 1990). We debated whether this study qualified for review. In particular, we discussed the issue with our pal senior content advisers, authors of a meta-analysis of pal studies. They had wrestled with the same issue, and agreed to support whichever decision we made. Based on our desire to provide educators as much information as possible on peer-assisted learning, we reviewed the study.
Regarding this, Mr. Pressley also questions the practice of writing our own summaries of reviewed studies. We believe this is important to provide our users with a standardized, highly accessible reporting mechanism. This approach builds on the advice of educator focus groups. Following our standard practice, we contacted the study’s primary author, Linda M. Lysynchuk, for feedback on the draft WWC report. She found it to be acceptable and requested no changes.
We encourage readers to reach us directly at info@whatworks.ed.gov, for we know that their comments, like Mr. Pressley’s, will help to increase the WWC’s value to decisionmakers in education.
Becki Herman
Project Director
Bob Boruch
Principal Investigator
American Institutes for Research
Washington, D.C.
