Report Offers ‘Index’ for Evaluating Ed-Tech

By Michele Molnar — July 23, 2013 3 min read
  • Save to favorites
  • Print

A report released today offers an “innovation index” meant to help school leaders and teachers, education company leaders, and investors determine whether digital innovations deliver on their promised classroom benefits.

The index allows evaluators to rate digital products and services based on three broad criteria: pedagogy, system change, and the quality of the technology.

Michael Fullan and Katelyn Donnelly co-authored the index, which is explained in detail in a new report titled “Alive in the Swamp: Assessing Digital Innovations in Education.”

The index is available as an “open source” tool to be used by anyone, free of cost. “We expect the biggest impact to be on system leaders making procurement decisions about which tools to introduce into the classroom,” said Donnelly, an executive director at Pearson, a major producer of educational technology and products, in a phone interview. Her co-author, Fullan, is a professor emeritus at the University of Toronto who now works to accomplish whole-system reform in educational systems.

Technologies are assessed within each of the three categories based on a series of questions. They are then scored on a color-coded system, which ranks them as follows:

  • Green innovations are educational technologies that are likely to succeed and produce transformative outcomes.
  • Amber green indicates an educational technology with some solid features, though also a few that have not reached their full potential.
  • Amber red is a problematic innovation, one that requires substantial attention because of shortcomings.
  • Red innovations are off track and unlikely to succeed.

“As an entrepreneur, you could go through every one of the buckets and give yourself a rating and self-assessment. How would you fill those gaps?” explained Donnelly, who also leads the Affordable Learning Fund, a venture fund that invests in early-stage companies serving schools and learners in the developing world. Investors wanting to weigh the potential impact of a technology can also gain insights from using the index, she said.

One of the ways the authors judge educational technologies’ potential to change systems is on their “value for money.” To get a “green” in that section, the authors write that the innovation “should be able to produce twice the learning outcome for half the cost of previous methods...The school and learner are significantly better off with the innovation and would actively choose to allocate scarce resources towards its purchase.”

The index addresses common challenges schools face in supporting new technologies, and issues they need to consider. For example, schools often overlook the importance of professional development when introducing new technologies, Donnelly said.

“To make those tools irresistibly engaging, first the technology has to be fantastic from a user-design perspective,” she said, “then it has to be integrated appropriately in the learning environment.”

When considering educational technology’s potential pedagogical benefits, users should consider factors such as the clarity and quality of the intended outcome, and the quality of the assessment platform and its functionality, the authors say. When considering the quality of technology, the index suggests they consider the design of the tech tool, its comprehensiveness, and how easily it can be adapted to different needs.

Donnelly said she has field-tested the index in North America with positive results. “We’ve trialed a couple of innovations, and it takes one and a half to three hours [to evaluate] each innovation. That assumes a baseline knowledge of the innovation and all its parts.”

The report was published by NewSchools, a U.S.-based nonprofit venture philanthropy firm working to transform public education for low-income children, and Nesta, a U.K.-based innovation foundation.

The authors are looking for feedback on the index from the business and education community. Donnelly believes the index ultimately could help combine the power of technology and instruction to bring improvements to classrooms. “If we can unlock pedagogy,” Donnelly said, “can we finally unleash the promise that the ed-tech revolution has in store?”

This item was originally posted on the Marketplace K-12 blog.

A version of this news article first appeared in the Digital Education blog.