School & District Management

Educators Wary of Ed-Tech Company Research

By Michele Molnar & Michelle R. Davis — August 28, 2018 6 min read
  • Save to favorites
  • Print

Most educators believe educational technology companies are not qualified to conduct valid research about their products, a new survey finds.

Yet the survey also found they rarely consult the What Works Clearinghouse—considered the “gold standard” by academic researchers—to determine the effectiveness of such products.

That puts them in a difficult spot, having to make decisions about the products without really knowing their efficacy.

Those are among the key findings and insights from the Education Research Perspectives Survey, recently released by the International Society for Technology in Education and the Jefferson Education Exchange, a nonprofit that aims to help educators and education leaders make better informed decisions about ed-tech.

In the nationally representative survey of 1,124 teachers, principals, district administrators, and technology leaders—60 percent of whom are ISTE members—76 percent felt that ed-tech vendors were not qualified to conduct valid research about their products, yet 48 percent still look at vendors’ sites to get information about those tools.

When he reviews research sponsored by companies, “I take it with a grain of salt,” said David Quinn, the director of technology integration for the 2,400-student Mendon-Upton Regional school district in Massachusetts. “It’s not that it’s invalid, but I understand that sometimes it is advocacy research.”

Natalie Makulski, a 3rd grade teacher at Botsford Elementary School in Livonia, Mich., who is finishing a Ph.D. in educational technology, considers vendor-provided research but wants to know whether the company’s study has been conducted by an independent researcher.

“I really want to know what the teacher does with it in their classroom, because that’s the real jungle,” she said. “But I do enjoy the company perspective as well.”

What Clearinghouse?

Makulski searches for studies on GoogleScholar. Yet she and Quinn do not regularly consult the What Works Clearinghouse, a repository of federally funded research under the U.S. Department of Education’s Institute of Education Sciences.

They’re not alone: 81 percent of this largely “plugged-in” group of educators does not regularly review research from the clearinghouse.

That isn’t news to Mark Schneider, the first permanent director of the IES in four years, who was confirmed in March.

“I’m not surprised by this information because I think the What Works Clearinghouse needs a serious face-lift, which we are in the process of doing,” he said. “The people we need to talk to are not researchers but practitioners.”

To make its findings more accessible, the organization is working on translating its contents into “plain language” and improving the search functionality on its website, he said.

“For companies, what passes as research is self-serving, and our goal is to figure out how to change that,” Schneider said.

Earlier this year, the Education Technology Industry Network, a division of the Software & Information Industry Association, released updated guidelines in an attempt to make its research more relevant to educators and other K-12 decisionmakers.

“Most research ends up not really benefiting anybody,” Mitch Weisburgh, the president of the industry association, and the managing partner of Academic Business Advisors, said when the ed-tech industry released new guidelines earlier this year. “We’ve been caught up with the same type of research pharmaceutical companies use. That research doesn’t work for education. It’s too big, too expensive, and it takes too long.”

So ETIN updated its best-practices document about conducting research to include considerations about usage, implementation issues, and findings for subgroups of students.

Purchasing Decisions

Despite the challenges, some research is making its way into ed-tech purchasing decisions with this largely tech-savvy group.

The survey respondents were asked about their involvement with procurement. Of those who are involved, 65 percent said that they discuss ed-tech research with colleagues when working on a committee that makes procurement decisions, and 57 percent discuss such research with colleagues when seeking approval to use a new tool or product.

That group is more likely to be involved with ed-tech procurement than a general group of educators would be—possibly because more than half of those who answered the survey are from ISTE, and 28 percent from ASCD, said Brandon Olszewski, ISTE’s director of research. The universe of K-12 educators is less likely to be involved with researching and buying ed-tech products.

Makulski, the Michigan teacher, said most of her education colleagues rely on word-of-mouth, not research of any kind, to choose products.

They “will buy these wonderful products, but when they get them, they don’t know what to do with them,” she said. “That’s the sparkle and glitter we have to get away from. They need to know the purpose behind what they’re buying.”

Quinn from Massachusetts tells a similar tale. Most teachers in his district rely on peer recommendations to make their own choices, he said.

His quest with researching products is to identify ones that drive achievement and also engage students. “We don’t want a tool that will increase scores but make students hate to read,” he said.

Finding a trusted resource for useful and accessible research is an ongoing challenge.

“We continue to spend billions on these products without the information we need to make good decisions,” said Bart Epstein, the president and CEO of the Jefferson Education Exchange.

‘There Is No Research’

Joseph South, the chief learning officer for ISTE, said he wants to change the dynamic in which ed-tech products are “chosen based on popularity, not efficacy.”

The co-sponsors of the Education Research survey are both working to address the issues raised in their research. For one, they are collaborating with other groups, like Digital Promise, a nonprofit that promotes the effective use of research and technology in education, to elevate the issue of sharing reliable information about ed-tech products.

ISTE also opened a community-driven review platform called the ISTE Edtech Advisor earlier this year. That resource is designed to give members of ISTE insight into which tools, technology, and apps are most likely to best meet their learning objectives.

And the Jefferson Education Exchange, launched in February, is a nonprofit organization working to improve how schools choose, procure, and implement educational technology. It plans to pay teachers for reviews of ed-tech products.

Epstein said further research is needed to understand more about how often educators consult peer-reviewed research, and what percentage of them do, to inform decisionmaking, too.

Quinn said he values both qualitative and quantitative aspects of research on digital products, but finds the quantitative metrics are often around very narrow aspects of usage. When those narrow measures are applied more broadly, “the conclusions can be a little exaggerated,” he said.

He has his own strategy for evaluating any ed-tech tool or app. First, he looks at “the most critical piece of feedback” about it, he said, then works backward from that point in establishing its potential value to teachers and students.

For Makulski, who sometimes wants to check out a recently released ed-tech tool, there’s another issue. “With a lot of the newer ones, you just have to take a risk,” she said, “because there is no research.”

An alternative version of this article appeared on EdWeek Market Brief.
A version of this article appeared in the August 29, 2018 edition of Education Week as Educators Wary of Ed-Tech Company Research

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Law & Courts Webinar
Future of the First Amendment: Exploring Trends in High School Students’ Views of Free Speech
Learn how educators are navigating student free speech issues and addressing controversial topics like gender and race in the classroom.
Content provided by The John S. and James L. Knight Foundation
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Start Strong With Solid SEL Implementation: Success Strategies for the New School Year
Join Satchel Pulse to learn why implementing a solid SEL program at the beginning of the year will deliver maximum impact to your students.
Content provided by Satchel Pulse
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Science Webinar
Real-World Problem Solving: How Invention Education Drives Student Learning
Hear from student inventors and K-12 teachers about how invention education enhances learning, opens minds, and preps students for the future.
Content provided by The Lemelson Foundation

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Schools Prefer Cheaper Ventilation Options to Curb COVID: Why They Should Consider Upgrading
Most schools are opening windows and hosting class outdoors rather than investing in costlier, more-effective measures.
2 min read
Students from PS 11 Elementary School participate in art projects and interactive activities, during an after-school outdoor program held in the High Line park in New York, NY, October 21, 2020.
Students from PS 11 Elementary School participate in art projects and interactive activities during an after-school outdoor program in New York City in 2020. Many schools are opting for outdoor classes and other-low cost measures to maintain healthy air quality during the pandemic.
Anthony Behar/Sipa via AP Images
School & District Management Hour by Busy Hour: What a Principal's Day Actually Looks Like
From the time they wake up until they set the alarm at night, school leaders juggle the routine, the unexpected, and the downright bizarre.
Left, Principal Michael C. Brown talks on a radio at Winters Mill High School in Westminster, Md., on May 17, 2022. Right, Boone Elementary School principal Manuela Haberer directs students and parents in the pick-up line at the conclusion of the school day on May 19, 2022 in San Antonio, Texas.
Left, Principal Michael C. Brown talks on a radio at Winters Mill High School in Westminster, Md., on May 17, 2022. Right, Boone Elementary School principal Manuela Haberer directs students and parents in the pick-up line at the conclusion of the school day on May 19, 2022 in San Antonio, Texas.
From left, Steve Ruark and Lisa Krantz for Education Week
School & District Management Photos What School Leadership Looks Like: A Day in the Life of a Principal
A look at a typical day for one elementary school principal in Texas and a high school principal in Maryland.
1 min read
Principal Michael C. Brown, from left, talks to seniors Brady D’Anthony, 18, and Sydney Dryden, 17, at Winters Mill High School in Westminster, Md., Tuesday, May 17, 2022.
Principal Michael C. Brown, from left, talks to seniors Brady D’Anthony, 18, and Sydney Dryden, 17, at Winters Mill High School in Westminster, Md., Tuesday, May 17, 2022.
Steve Ruark for Education Week
School & District Management Schools Can Access Tons of Money for Electric Buses. Will They Use It?
Electric buses are growing more appealing as fuel prices rise, but some districts remain wary of the cost and logistics.
5 min read
Stockton Unified School District's new electric bus fleet reduces over 120,000 pounds of carbon emissions and leverages The Mobility House's smart charging and energy management system.
The new electric bus fleet at California's Stockton Unified School District is projected to reduce the district's carbon emissions.
Business Wire via AP