Curriculum

Textbooks Alone Don’t Boost Test Scores, Study Says

By Stephen Sawchuk — March 11, 2019 6 min read
BRIC ARCHIVE

Replacing a math textbook is unlikely to generate large boosts in student achievement on its own, concludes a new study that stands in contrast to earlier research suggesting stronger benefits from overhauling learning materials.

In results even the researchers struggled to interpret, none of 14 math textbooks studied was consistently linked to gains in 4th and 5th grade student test scores collected from hundreds of schools in six states. They also found few differences in student learning between textbooks written before the Common Core State Standards were put into place in those states and those textbooks purportedly designed to match the standards.

The findings don’t imply that textbooks are interchangeable, since the study did not specifically address the relative quality of the materials. But the study does hint that textbook effectiveness may be closely tied to how the books are adapted and used by teachers, and probably how well teachers are trained to use them.

It’s also possible that some of the textbooks did show smaller effects that the study couldn’t detect.

In all, the researchers underscored that district officials should still consider things like content alignment when making curriculum decisions—but such factors may only go so far when it comes to their impact.

“Using a better frying pan doesn’t necessarily mean you make a better omelet,” said Thomas Kane, a professor of education at Harvard University and the faculty director of the Center for Education Policy Research, which conducted the study, released on Monday. “We were chasing the omelets, rather than looking at the other aspects. And given the way textbooks are being used, we’re not seeing substantial differences.”

Tough to Study

For something that affects nearly every classroom in the nation’s more than 14,000 districts, there is little empirical research linking particular learning materials to student gains.

Mostly, that’s a data-collection problem: Very few states require districts to report which materials they choose for each grade or subject, and publishers jealously guard market-share information, which means tracking down districts’ choices is time-consuming and difficult. In fact, most curriculum studies have been small in scale or focused on individual textbooks, thus producing results that aren’t easily generalized to the student population at large.

Two recent studies have made researchers perk up, though: one randomized study in 2016 on a math series, and another analysis on a variety of California math textbooks, in 2017. Both linked certain books to moderate-to-large gains in student learning.

In the new study, a team of 11 researchers set out to see if they could replicate those findings with a larger, multistate sample. They collected data on math textbooks from classes in California, Louisiana, Maryland, New Jersey, New Mexico, and Washington state and linked it to the test scores of students taught using those books from 2014-15 through 2016-17. (All those states adopted the common core and related tests over the last decade.)

The researchers used statistical methods to look at the data, slicing and dicing it in a few different ways. Some models analyzed only one year of test-score data, others looked at pooled years of data. One model took out California, which had by far the most schools in the sample, and another looked only at that state.

They found links between some of the books and student achievement for some of the models but not for others. No textbook was consistently linked to student learning.

It’s possible that data limitations prevented the researchers from capturing smaller, more nuanced effects, said Cory Koedel, an associate professor of economics and public policy at the University of Missouri. That’s because the sample may have lacked the statistical power to discern smaller effects.

An appendix to the report shows that a curriculum with a 5 percent market share would not produce a statistically significant effect size in the data unless it approached .10 of a standard deviation—a size considered moderate. Of the textbooks studied, many did not have a market share that size.

“Finding an effective textbook is like looking for a needle in a haystack,” said Koedel, who reviewed earlier drafts of the research but did not contribute to it. “I think [these researchers] have mostly ruled out large effects from a textbook, but moderate-to-small effects are still on the table.”

And that poses a research quandary, Koedel said, because even small effects are potentially significant when it comes to textbooks. After all, he said, textbooks affect all students, and changing them costs much less than other education interventions that require hiring more teachers, or that impact only some students.

“You have to buy a book anyway,” he pointed out. “If you take it in that context, a modest effect can be pretty important.”

Researchers also offered another hypothesis for the null results: Teachers don’t use the books consistently enough or don’t receive enough training on them. In a survey of teachers in a subset of the schools, nearly all reported using their textbooks to some extent, but only a quarter said they relied on them to guide most teaching, homework assignments, and exams. Only 7 percent said they exclusively used the book materials.

Questions of Use

Instead, teachers supplemented with other materials, most frequently with state or local resources, online lesson plans, or supplies created by other teachers. Some said that the textbooks were too hard or easy for their students; over half said that the textbook examples were “not sufficiently engaging” for students. Also, few teachers received more than a day or two of training on their textbooks.

“Honestly, I think that’s the most important next question,” Kane said. “What levels of coaching or more-intensive professional development are required to help teachers use rigorous materials at higher levels of fidelity, and does that produce larger benefits?”

The norms surrounding teachers’ attitudes toward materials—and how that can make or break implementation—are only starting to be investigated in any depth.

Emily Freitag is the CEO and co-founder of Instruction Partners, a nonprofit that works with schools to improve academics. After interviewing educators in 75 districts about their curriculum-adoption processes, her group found marked differences in attitudes towards curriculum between those school systems that put materials at the center of their vision for teaching, learning, and grading, and those that mostly did it out of compliance. The group has since put out a free curriculum-support guide for districts.

“What we consistently hear from teachers is, ‘Tell me what good looks like. Because that’s what I want to do.’ And I have seen a great curriculum when supported well absolutely transform schools, because I think it answers that question for everyone,” said Freitag, who had not yet reviewed the Harvard research. “And I have seen those same materials, though, fall flat. So I don’t think the materials alone can be our obsession. I think how we support great instruction needs to be the obsession.”

The Harvard research was supported in part by the Bill & Melinda Gates Foundation, as was Instruction Partners’ guide. The philanthropy has provided funding to Education Week in the past, but does not do so currently.

Related Tags:

A version of this article appeared in the March 13, 2019 edition of Education Week as New Texts Failed to Lift Test Scores in Six-State Study

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Measuring & Supporting Student Well-Being: A Researcher and District Leader Roundtable
Students’ social-emotional well-being matters. The positive and negative emotions students feel are essential characteristics of their psychology, indicators of their well-being, and mediators of their success in school and life. Supportive relationships with peers, school
Content provided by Panorama Education
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
Making Digital Literacy a Priority: An Administrator’s Perspective
Join us as we delve into the efforts of our panelists and their initiatives to make digital skills a “must have” for their district. We’ll discuss with district leadership how they have kept digital literacy
Content provided by Learning.com
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School & District Management Webinar
How Schools Can Implement Safe In-Person Learning
In order for in-person schooling to resume, it will be necessary to instill a sense of confidence that it is safe to return. BD is hosting a virtual panel discussing the benefits of asymptomatic screening
Content provided by BD

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Whitepaper
Educator Survey Results: Meeting the Demands of Hybrid Learning with eBooks
With COVID-19 altering nearly all aspects of daily life, including the way students learn, this survey sought insight from those on the f...
Content provided by OverDrive
Curriculum Opinion Introducing Primary Sources to Students
Five educators share strategies for introducing primary sources to students, including English-language learners.
12 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
iStock/Getty
Curriculum Opinion Eight Ways to Teach With Primary Sources
Four educators share ways they use primary sources with students, including a strategy called "Zoom."
13 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."
iStock/Getty
Curriculum The Dr. Seuss Controversy: What Educators Need to Know
The business that manages Dr. Seuss' work and legacy will cease publishing six books due to racist stereotypes and offensive content.
5 min read
A copy of the book "And to Think That I Saw It on Mulberry Street," by Dr. Seuss, rests in a chair on March 1, 2021, in Walpole, Mass. Dr. Seuss Enterprises, the business that preserves and protects the author and illustrator's legacy, announced on his birthday, Tuesday, March 2, 2021, that it would cease publication of several children's titles including "And to Think That I Saw It on Mulberry Street" and "If I Ran the Zoo," because of insensitive and racist imagery.
Dr. Seuss Enterprises announced it would cease publication of several of the author's children's titles because of insensitive and racist imagery.
Steven Senne/AP