Executive Skills & Strategy

New Tools Seek to Evaluate Ed-Tech Products

By Jason Tomassini — October 16, 2012 7 min read
  • Save to favorites
  • Print

A number of services have cropped up over the years to help schools answer a question that, in an age of information overload, scarce resources, and new technology, is becoming central to how they move forward: What works?

Whether it’s aggregating the most trustworthy studies on education or culling user reviews on products, as Yelp does with restaurants, none of these available services, both privately and publicly financed, has gained national scale. But a new proposal from two economists is perhaps the most ambitious—or, if you ask some of its potential users, flawed—attempt at creating a Consumer Reports for education technology to address the lack of independent evidence available on such products’ efficacy.

The proposal is called Edu Star, a technology tool that would allow schools to conduct rapid, randomized evaluations of education products, collect and analyze the results, and publish the data to the public. By doing so, schools would make better-informed purchasing decisions, and entrepreneurs would have evidence that their products worked, the economists hope.

“Not surprisingly, when no one knows what works, schools are unlikely to buy, and innovators are unlikely to create,” the economists, Aaron Chatterji, a professor at Duke University, and Benjamin Jones, a professor at Northwestern University, write in a discussion paper sponsored by the Hamilton Project, an economic-policy initiative of the Brookings Institution. The paper was discussed at a Sept. 27 forum hosted by the Washington think tank.

How Edu Star Would Do Evaluations

A new proposal from two economists would create a Web-based system to rapidly conduct randomized trials among students on the efficacy of education technology products.


SOURCE: The Hamilton Project, “Harnessing Technology to Improve K-12 Education”

Innovate Faster?

Mr. Chatterji and Mr. Jones came up with the idea while both were on the White House Council of Economic Advisers two years ago. Edu Star is a convergence of three factors they believe drive much of the economy’s health: education, innovation, and “building things,” as Mr. Jones put it in an interview.

“If there is a weak link in this interconnected chain, it really seems to be K-12 education,” he said.

K-12 education accounts for just 0.2 percent of the research and development expenditures in the United States, according to a 2010 report by the President’s Council of Advisers on Science and Technology, despite $600 billion in total annual K-12 expenditures. A low-cost way to conduct randomized trials could help education technology innovate at the same speed as other fields, where products are rolled out and tested at a much faster rate, Mr. Jones said.

To understand how Edu Star’s trials would work, Mr. Chatterji and Mr. Jones suggest thinking of Google’s rapid product development rather than, say, the U.S. Food and Drug Administration’s trial studies. Education technology companies would offer their products to be tested, and schools would agree to test the products. After logging in to Edu Star, students would take a pre- and post-assessment on a skill tied to the Common Core State Standards.

In between, students would be assigned exercises in either one of two products built to teach that skill or a placebo product. All of the results would be recorded in Edu Star and reported on its website.

Schools would have an idea of whether a product improves academic outcomes, proponents of the plan say. Entrepreneurs without major sales forces could shift focus from wooing district leaders to sign contracts to proving a product’s value to its end user, the teacher, they say.

“It’s not an enterprise sale, it’s the bottom-up approach,” Eric Westendorf, the chief executive officer of LearnZillion, a Washington-based video-lesson-sharing tool, said at the recent forum. “That, working in collaboration with an Edu Star-type platform, is really exciting.”

But early responses to the proposal suggest Edu Star takes a limited view of what’s required to evaluate educational products. Factors such as professional development, ease of implementation, and long-term academic performance are also important to purchasing decisions, said Mark Edwards, the superintendent of the 5,500-student Mooresville, N.C., school district.

“In my experience, there’s a level of complexity around using digital resources that would not align with a quick and efficient means of evaluation,” he said.

Other Services Emerge

A commonly referenced barrier for companies in the education market is the country’s approximately 14,000 school districts, which essentially make up 14,000 distinct buyers. But many of those barriers are reduced because technology companies can enter the market cheaply.

The common-core standards, adopted by all but four states, are likely to both simplify the market for vendors and create confusion for school-level customers.

“Almost everything you see now says, ‘aligned with common core.’ I don’t believe that,” Dennis Van Roekel, the president of the 3.2 million-member National Education Association, said at the Washington forum. “But everyone says so because it’s a nice marketing line.”

Some services have cropped up to help make sense of the market, and most are geared toward teachers, indicative of their increasing role in the marketplace.

EdSurge, a Silicon Valley startup news service that covers the education technology industry, offers selective reviews of digital learning tools from educators who have used them. EdShelf, another Silicon Valley-based company, offers a directory, reviews, and videos of digital learning tools.

PowerMyLearning, a K-12 learning platform from CFY, or Computers for Youth, a New York City nonprofit organization, rates free digital educational content using reviews from teachers, students, and parents. The organization is in talks with Edu Star about potential partnerships, said CFY co-founder Elisabeth Stock. Using both anecdotal user reviews and empirical randomized trials would help give a fuller picture of the education technology marketplace, she said.

“You can have a teacher who uses a product and has great results, but that doesn’t mean that teacher has tried that product against three other products,” Ms. Stock said. (The Bill & Melinda Gates, Eli and Edythe Broad, and W.K. Kellogg foundations contributed a total of $7 million to CFY. The Gates Foundation provides grant support for Education Week‘s coverage of K-12 business and innovation.)

Another effort, the Learning Resource Metadata Initiative, funded by publishing associations and foundations, including Gates, aims to categorize the open learning materials available on the Web for easier search.

The What Works Clearinghouse, a website funded by the U.S. Department of Education, evaluates the credibility of education research. But there are few up-to-date items on education technology on the site and, overall, it’s been criticized for a lack of user-friendliness.

‘The Right Answers’

But would Edu Star itself work?

Right now, it is only a proposal, though it’s one that Mr. Chatterji and Mr. Jones intend to pursue, by raising $5 million in foundation funding and hiring a small staff. (Neither plans to work on Edu Star full time.)

The idea has drawn early critics, meanwhile.

Betsy Corcoran, EdSurge’s chief executive officer, doesn’t believe the Edu Star ratings would work for modern technology products constantly in flux. “It won’t accurately capture the usefulness of young and emerging tools because those tools will be in transition a lot,” she said.

Mr. Westenberg of LearnZillion said it is “scary” that a rating of an early version of his product could be published.

There is also criticism that the rapid assessments the proposal envisions aren’t the best way to judge a product, let alone student learning in general.

“Ah, educational research. Ah, test scores. Ah, common core. Ah, what a very limited definition of ‘learning,’ ” wrote education technology blogger Audrey Watters.

Mr. Jones, the Northwestern economist, argues that because Edu Star would test students on the common standards, what was being assessed would have legitimacy and would provide an apples-to-apples way of judging disparate groups of students.

But in general, he acknowledged, there is still a lot to figure out, including how to provide incentives for participation for entrepreneurs and schools, and how to navigate the Family Educational Rights and Privacy Act, which protects the confidentiality of students’ education data.

Mr. Jones also stressed that the proposal doesn’t aim to reinvent learning or the education marketplace; it’s just another tool for helping produce better decisionmaking.

“The real innovation isn’t Edu Star,” he said. “It’s not about knowing what the right answer is, it’s about creating a system where the right answers can emerge.”

A grant from the Bill & Melinda Gates Foundation helps support coverage of the education marketplace and new approaches to schooling in Education Week and on edweek.org.
A version of this article appeared in the October 17, 2012 edition of Education Week as New Tools Seek to Evaluate Ed-Tech Products, Services


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Classroom Technology Webinar
Transform Teaching and Learning with AI
Increase productivity and support innovative teaching with AI in the classroom.
Content provided by Promethean
Curriculum Webinar Computer Science Education Movement Gathers Momentum. How Should Schools React?
Discover how schools can expand opportunities for students to study computer science education.
School & District Management Webinar Fostering Student Well-Being with Programs That Work
Protecting student well-being has never been more important. Join this webinar to learn how to ensure your programs yield the best outcomes.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Executive Skills & Strategy Opinion Stop Ignoring the Innovation That Happens in Traditional Public Schools
Three national educational funders explain a new program that is highlighting innovative practices in schools around the country.
Jenny Curtin, Britt Neuhaus & Saskia Levy Thompson
4 min read
Executive Skills & Strategy 'Genius Hour' Lets Kids Take Charge: Would Einstein Have Liked This?
Teachers open doors for students to fuel their curiosity and pursue passion projects, but educators warn against making it a free-for-all.
10 min read
Quin, Ezra, and Owen participate in genius hour in teacher Melisa Hayes’ 2nd grade class at Avery Elementary School in Hilliard, Ohio.
Quin, Ezra, and Owen participate in genius hour in teacher Melisa Hayes’ 2nd grade class at Avery Elementary School in Hilliard, Ohio.
Maddie McGarvey
Executive Skills & Strategy K12 Inc., Ga. Cyber Academy Contract Battle Brews
Students locked out of their school's computer systems. Educators unable to get access to some students' records. Parents receiving emails asking that they return their children's laptops.
6 min read
Executive Skills & Strategy Report Roundup Research Report: Teaching
Forty percent of what elementary school teachers do on a typical workday could be automated by 2030, predicts a new report by the McKinsey Global Institute.
1 min read