Corrected: A previous version of this story cited an incorrect figure for the share of K-2 early reading and special education teachers who said they use Fountas and Pinnell’s Leveled Literacy Intervention.
Two of the nation’s most popular early literacy programs that have been at the center of a debate over how to best teach reading both faced more new critiques in the past few weeks, receiving bottom marks on an outside evaluation of their materials.
EdReports—a nonprofit organization that reviews K-12 instructional materials in English/language arts, math, and science—published its evaluation of Fountas and Pinnell Classroom Tuesday, finding that the program didn’t meet expectations for text quality or alignment to standards. The release comes on the heels of the group’s negative evaluation last month of the Units of Study from the Teachers College Reading and Writing Project, another popular early reading program.
Together, the two reports received the lowest ratings EdReports has given for K-2 curricula in English/language arts, and they’re among the three lowest for ELA in grades 3-8.
“The materials don’t reflect the shifts—text quality and complexity—especially in K-2,” said Stephanie Stephens, EdReports’ ELA content specialist for early literacy, referencing key components of the Common Core State Standards—a big part of the organization’s review criteria.
These two literacy brands, both published by Heinemann, command large shares of the early reading market.
In 2019, a nationally representative EdWeek Research Center survey found that 43 percent of K-2 early reading and special education teachers use Fountas and Pinnell’s Leveled Literacy Intervention, the intervention companion to Fountas and Pinnell Classroom. The same survey found that 16 percent of teachers used the Units of Study for Teaching Reading.
Recently, these programs have faced criticism from educators and researchers that the instructional methods they use don’t align with, or in some cases contradict, the research on how to develop strong readers. Fountas and Pinnell has pushed back against these characterizations. Lucy Calkins, the director of the Teachers College Reading and Writing Project, has announced an upcoming revision to the Units of Study, set to be released in summer 2022. (EdReports reviewed the current version of the materials.)
How these programs attend to foundational skills—teaching students to recognize and manipulate the sounds in words, and then matching those sounds to written letters—is one of the main focuses of the critique. It’s also something that EdReports turned a renewed attention toward.
Fountas and Pinnell Classroom and Units of Study are two of the three K-2 reading programs to have gone through EdReports’ updated review tools for English/language arts, which “dig deeper” into the sequencing of foundational skills teaching. These new evaluation criteria also look for what EdReports calls “bloat,” whether all of the content in a set of materials can be taught in one year. Open Court, the third program evaluated with these new tools, partially met expectations.
In its two responses to the reviews on the EdReports website, Heinemann wrote that the EdReports’ rubrics aren’t a good fit for programs like Fountas and Pinnell Classroom and Units of Study.
“FPC greatly values the importance of responsive teaching and the teacher agency required to adjust, extend, and enrich learning based on individual student needs,” reads one response. “The EdReports rubric provides no way to measure these deeply valuable components of an effective literacy system.”
This ethos of teacher agency is one of the reasons that Fountas and Pinnell and the Units of Study are so popular with educators. Both give teachers, and oftentimes students, choice over materials and activities. Still, the EdReports reviews could affect whether schools continue to use them, said Morgan Polikoff, an associate professor of education at USC Rossier who studies K-12 curriculum and standards.
The reviews may influence state-level recommendations, or district leaders might reference them the next time they have to choose curriculum, Polikoff said. It’s also possible that parent advocates, like those in Minnesota who have petitioned their school board for better reading instruction, could use these reviews. “A bad EdReports rating could be another piece of evidence that those parents could potentially bring to bear” in attempts to jettison these programs, he said.
Kareem Weaver, a member of the Oakland NAACP Education Committee and the co-founder of FULCRUM, an Oakland group that advocates for evidence-based literacy instruction, said that the reviews could provide the impetus for school districts to reconsider the use of programs that he says don’t work for all kids.
“I’m really hoping it will make people do a double take,” he said.
Reviews critique text complexity, foundational skills
Since its launch in 2015, EdReports has recruited educators—teachers and other instructional leaders—to conduct its reviews, and to develop the rubrics used to judge materials. These rubrics measure alignment to the Common Core State Standards, usability in a classroom setting, and other indicators of quality, such as text complexity.
The company is one of the few organizations that provides external evaluations of curricula, and its reviews have a wide reach: As of 2020, EdReports said that at least 1,084 districts use its reviews, including 89 of the 200 largest districts in the country. (There are about 13,400 school districts in the United States.)
Still, not everyone agrees with EdReports’ conclusions. Publishers have critiqued the group’s methodology and rating system in the past, claiming that reviewers failed to consider supplemental materials and taking issue with the organization’s “gateway” system, which requires that a program meet the standards set for alignment before it can be evaluated on other features. EdReports made a few changes to its process after publishers pushed back on its first set of math reviews, though the gateway system remains.
Fountas and Pinnell Classroom failed to pass the first gateway. In K-2, reviewers said that core texts didn’t meet standards for quality or complexity, and that speaking and writing assignments didn’t require students to use evidence from the texts they read. EdReports also critiqued Fountas and Pinnell’s text leveling system, which it said was “not accompanied by an accurate text complexity analysis and a rationale for educational purpose and placement in the grade level.” The group gave a similar evaluation for the program in grades 3-8.
While the K-2 program’s word study lessons teach phonics, “the program does not present a research-based or evidence-based explanation for the sequence” of instruction, reviewers found. The report also claims that the program doesn’t consistently devote enough time to systematic instruction in phonological awareness, phonics, and fluency.
Units of Study also didn’t pass EdReports’ first gateway, which measures alignment to the common core. For grades K-2, reviewers said that texts featured in the materials “are not appropriately complex for the grade level and do not build in complexity over the course of the year.” They also noted that the program focused mostly on reading skills instruction, rather than “questions and tasks aligned to grade-level standards,” like asking students to use information from the text to support opinions.
Instruction in foundational reading skills like phonological awareness and phonics, they said, “lacks a cohesive and intentional scope and sequence.” The review also notes that the materials rely on cueing strategies for word identification: prompting students to draw on pictures, context, and sentence structure—along with letters—to figure out what words say. But research has shown that pulling students’ attention away from the letters can lower the chances that they’ll use their knowledge of letter-sound correspondences to read through a word, making it less likely that they’ll be able to map the spelling to the spoken word in their memory.
Reviewers found text complexity lacking in grades 3-8, as well, and they said that the program lacks “a variety of regular, standards-aligned, text-based listening and speaking opportunities,” as well as opportunities for on-demand writing and systematic vocabulary development.
Not every program reviewed against EdReports’ new rubric received low marks. Open Court, the third program reviewed with the new tools, fared better. It partially met expectations at the first gateway, and also at the second gateway, which measures knowledge building. In grades K-2, reviewers reported a research-based approach to foundational skills instruction, but noted that there wasn’t enough practice with encoding—hearing sounds and converting them into written language.
Reviewers said that only some texts were “appropriately complex for the grade level,” and also said that the program missed opportunities for standards-aligned activities. They had the same critique for the grades 3-5 materials.
Publishers claim that EdReports tool is mismatched to their approach
As part of the review process, EdReports solicits publisher responses to its evaluations, posted publicly on its website. McGraw Hill, which publishes Open Court, and Heinemann both critiqued the review process in their responses.
The McGraw Hill response claimed that EdReports had overlooked end-of-unit opportunities for students to demonstrate knowledge, citing the curriculum’s unit-long “Inquiry” process.
Heinemann criticized the EdReports review process for omitting texts that students read outside of whole-group instruction.
In Units of Study, students only spend limited time in a whole-group “minilesson,” before moving on to the reading workshop, during which they apply the skills taught in the minilesson to independent reading, reading with a partner, or working with the teacher one-on-one or in small groups. FPC is structured similarly, with whole-class minilessons but also guided reading, independent reading, and student book clubs.
Heinemann’s responses argue that EdReports’ review design prioritizes textbook-style reading curricula, and fails to capture the quality of texts that students might read on their own or in small groups. The publisher did not respond to EdReports’ critiques of foundational skills instruction.
Stephens, of EdReports, said the group is not discriminating based on design and approach, but rather evaluating whether students have guaranteed access to grade-level text. “If they’re using independent reading at their level, there’s not a guarantee that’s at grade level,” she said.
Separately, Irene Fountas and Gay Su Pinnell, the program’s namesakes and founding authors, have begun to publish a 10-part blog series rebutting claims that their program is not aligned to reading science.
In the series, the authors defend their program’s use of cueing and other strategies that are central to their materials but which studies have shows are ineffective, like leveled reading groups.
“If a reader says ‘pony’ for ‘horse’ because of information from the pictures, that tells the teacher that the reader is using meaning information from the pictures, as well as the structure of the language, but is neglecting to use the visual information of the print,” one of the blogs reads. “His response is partially correct, but the teacher needs to guide him to stop and work for accuracy.” This idea is in direct contrast to what most cognitive scientists say about how strong readers process new words.
The Teachers College Reading and Writing Project, which writes the Units of Study, has also separately responded to the EdReports reviews. A post on the group’s website argues that the program has a different approach to meeting common-core standards than EdReports does. “At a fundamental level, ours is a paradigm where choice matters, where agency matters. EdReports uses a rubric that does not value those things.” TCRWP cited, for example, that when teachers were provided with a choice to assign on-demand writing, EdReports didn’t award full marks because the writing was not a requirement.
“This is always the challenge of applying a rubric to things that differ in a lot of ways. It’s an imperfect science,” said Polikoff, of USC Rossier. “The question is, is it better than not having it? And to me, the answer is yes.”
EdReports is working with one set of criteria, and can give teachers information about how programs line up according to that criteria—information that is often hard to come by, Polikoff said. There aren’t many avenues for teachers to find third-party evaluations of materials, he added.
Matthew Alexander, the director of elementary literacy and numeracy for Hall County Schools in Gainesville, Ga., said his district relies both on outside evaluation and internal data in making decisions about what programs to use.
Hall County uses one piece of Fountas and Pinnell Classroom—the Phonics, Spelling, and Word Study component—across its 20 elementary schools. The district also use its Benchmark Assessment System.
Alexander plans to discuss the review with other leaders in the school system, as it relates to their phonics instruction. But he’s hesitant to make any quick changes, because Hall County only started using Phonics, Spelling, and Word Study in the 2019-20 school year, right before the pandemic hit.
“If we were seeing that in our schools, that our kids were not making gains as readers, we would certainly look to see if we would shift our resources in a different direction. But with just three years of non-typical data, it’s hard to make that statement,” Alexander said.
Review tool changes address foundational skills, program ‘bloat’
The low ratings on some indicators in these reviews stem from changes to EdReports’ review tools.
In 2020, EdReports announced its first revision to its criteria and its evidence guides—a sort of handbook for reviewers that helps them identify evidence that programs meet, or don’t meet, the criteria. Part of this update are two key changes to how reviewers evaluate English/language arts materials.
One has to do with how reviewers approach foundational skills instruction in K-5. Criteria and evidence guides are more specific about when and how these skills should be taught.
For example, criteria that require systematic and explicit teaching in the alphabetic principle, phonemic awareness, phonics, and other skills has now been split into four subcategories, each with its own grade-by-grade breakdown of what students should be able to do in the evidence guide.
EdReports has also cut guidance that says programs “should instruct the teacher to employ syntactic or semantic cueing systems when the phonics patterns do not work or to confirm a word choice.” These changes have come as reporting and the work of reading researchers have turned increased public scrutiny toward cueing over the past few years.
The revision brings the comprehensive ELA reviews more in line with the stricter criteria in stand-alone reviews of foundational skills, which EdReports launched in 2019, said Stephens. This way, she said, comprehensive reading programs will be judged as rigorously on their foundational skills components.
Still, Stephens thinks that the programs reviewed under the revised tools would have fared similarly under the originals. The revision provided “clarity,” she said, rather than an entire new scoring system.
The other change to the review process concerns what EdReports calls program “bloat.”
If a program says, for example, take 15 minutes a day for reading and 20 minutes for foundational skills, is that actually doable with the materials provided? Or is there too much content to feasibly get through? The program should offer a “clear and concise” pathway through the standards, Stephens said.
EdReports has also made some changes to its math review process, and has updated its criteria for gateway 3, which measures usability, across all subjects.
Louisa Moats, an early literacy expert and the lead writer of LETRS, a professional development program for reading teachers, has critiqued EdReports’ criteria in the past. She said that the new review tools are more closely aligned with research-based practice in reading instruction.
“These standards are much better for identifying practices in programs that are wildly off base. They’re a pretty good firewall in recognizing the programs that are wildly misaligned with reading science and with practices that have been shown to be ineffective with most kids. … That’s really good,” she said.
Still, she said, even if a program passes the review, its success or failure is going to come down to how the skills are taught in the classroom.
Weaver, in Oakland, said that the field needs more information about the effectiveness of popular reading materials. “What [EdReports] doesn’t do is it doesn’t talk about student achievement results. It doesn’t talk about how kids do with the program. And that’s fine, because they don’t claim to do that. But a lot of districts think they do,” he said.
“Alignment with the standards is the bare minimum that we should be able to expect from the curriculum,” Weaver said.
