Special Report
Personalized Learning

‘Red Flags’ to Look for When Evaluating Personalized Learning Products

By Michele Molnar — October 18, 2016 6 min read
  • Save to favorites
  • Print

Personalized learning: Is it an educational imperative, a marketing strategy for an ed-tech product, or both?

Too often, teachers and administrators say, they find that personalized learning is used by companies as mere buzzwords to promote a run-of-the-mill digital tool.

“In the marketing literature, this term is overused,” said Devin Vodicka, the superintendent of the Vista, Calif., school district. “Many products that someone claims are personalized are actually just a series of digital worksheets.”

But educators are finding ways to sort the real personalized potential from the empty promises of some ed-tech products.

For instance, Vista uses its own Personal Learning Pathway framework to evaluate products, said Vodicka.

Some of the key questions educators in his district ask:

• Is this product based on a student profile?

• Is there an integrated technology component that includes two-way communication?

• Does the product include student choice and pathways?

• How does it fit with the learning environment, and more broadly, is it connected with real-world opportunities?

• Does it use a competency-based model, in which students move at their own pace as they master academic content?

Last month, Vista hosted 42 superintendents from across the country, who discussed that and other approaches as part of an ongoing cohort studying personalized learning through AASA, The Superintendent’s Association.

The most important starting point to vet a product, for Barton Dassinger, the principal of Chavez Elementary in Chicago, is this query: Does this product improve student learning better than the alternative to using it? And does the company generate reports in such a way to allow for an analysis that will answer that question?

The starting points of “what is our need?” and “what is our goal?” are important for Théa Williams, who is both a technology teacher for pre-K-5 and the technology coordinator for Brooklyn Arbor Elementary School in New York City. As an iZone pilot-team leader through the city’s department of education, she said she has learned to move beyond the catchphrases for products to identify the data needed.

Red flags about the “personalized learning” label abound, according to Amy Nowell, the director of research for LEAP Innovations, a Chicago-based nonprofit that works to build educational innovations by connecting schools and businesses. From her experience testing ed-tech products with evidence-based research methods, Nowell, and other experts have identified several red flags educators should watch for:

Questionable Student Agency

Buzzwords in the Marketplace

This is a fundamental element of personalization. Can the students take ownership of their own learning by setting their own goals? Can they track their own progress?

Nowell recommends that teachers “test drive” products as though they are students and make sure they understand what the data mean.

For instance, if students aim to learn a particular unit within a specified period of time, and the product provides feedback that “you have only completed one activity” in that time frame, what, she asked, does that mean to a student trying to complete and comprehend the full unit?

Inadequate Content

By definition, personalized learning allows students to move at their own pace through material.

“There’s never a classroom where every student is average,” said Nowell. “We’ve had a number of teachers who were really disappointed when they dug in, and two months into the year, their brighter kids are looking for material that wasn’t there.”

While ed-tech products can often be retrofitted to accommodate material for advanced or struggling students, that’s often a “clunky” solution, she said. A digital resource with “only 130 lessons” is unlikely to have enough content to go up or down two grade levels.

Useless Data

Products for personalized learning generally produce a lot of data. Williams recommends educators first ask: “How do I make sense of this data?”

In theory, experts say a data dashboard should help students and teachers understand what the metrics are, how a certain metric was arrived at, and what it means for student learning. But that is not always the case.

For example, one product reported to teachers the percent of total lessons each student completed in the 4th grade curriculum. “Johnny has completed 4 percent of the lessons has no actual meaning to anybody,” Nowell said. “It’s not tied to what students are learning. It’s not tied to learning standards or mastery of content. Johnny could have clicked through, ‘completed’ them, and gotten them all wrong.”

Lacking Recommendations

“I need the data to help make instructional decisions,” said Williams.

The problem is that automating the process of using data to inform educators’ decisionmaking is still largely an unmet need. “I don’t feel personalized ed-tech products have mastered that yet,” she said. “I’d like to see more automation, providing data to teachers to make instructional interventions in the learning path.”

Poorly Aligned Assessments

Generally, the embedded assessment questions in personalized-learning products have had no external validation, Nowell said.

When piloting their products, companies want to know what assessments students will take, whether it’s from one of the common-core-aligned testing consortia or another well-known test. But because educational technology generally doesn’t have the resources for rigorous assessment design and validation, the assessment data that are generated from the trials might not align with the tests that eventually will be given to students, she said.

Classroom-Integration Problems

A company should be able to inform teachers how to best integrate their products into the classroom structure. Does the product work best in classrooms where groups of students rotate from one station to another, while the teacher instructs another small group? Can students work on it together, or solo?

“We’ve had teachers who started with the station-rotation model but had to revert to the typical classroom model,” said Nowell, “and it was really about the way the technology was set up, not about the kids not being mature enough to work that way.”

Another question is, “Who assigns the next piece of content? The teacher or the software?” Teachers want both options, she said.

Little Evidence It Works

Nowell cautions teachers not to put too much stock in testimonials from other users of a product, five-star ratings on websites, or anecdotes about how a product is used. “Press companies for hard data about documented outcomes,” she advised.

“Ask them for any studies done on how well their products impact student outcomes,” she said. For instance, companies should be able to provide empirical data on product effectiveness as it relates to student achievement. “Even case studies, if done well and based on a similar context to your classroom, can provide a powerful indication of how a product could potentially work for you.”

Measures of student learning can include nationally standardized test scores, as well as more timely measures of student engagement and motivation for learning, she added.

Lacking the Personal Perspective

Personalized learning means taking each child’s uniqueness into account.

“There are all these factors you have to consider: culture, family background, interests, learning strengths, social-emotional development. We could go on and on,” said Williams, of Brooklyn Arbor in New York. “We’re talking about a human being.”

Teachers would like to see products that provide them with more in-depth insights into their students.

Williams said she sees educators’ excitement about the prospect of personalization. But personalized-learning experts generally agree that most ed-tech products are not geared toward students’ individual backgrounds and interests.

“When they do find products that really work for them, and respond to how students answer questions then make adjustments accordingly, teachers can be so much more effective,” she said. “Then they can do the things that the personalized learning tool can’t do and focus their time and energy on that.”

Coverage of the implementation of college- and career-ready standards and the use of personalized learning is supported in part by a grant from the Bill & Melinda Gates Foundation. Education Week retains sole editorial control over the content of this coverage.
A version of this article appeared in the October 19, 2016 edition of Education Week as 8 ‘Red Flags’ to Look for in Products


Special Education K-12 Essentials Forum Innovative Approaches to Special Education
Join this free virtual event to explore innovations in the evolving landscape of special education.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
STEM Fusion: Empowering K-12 Education through Interdisciplinary Integration
Join our webinar to learn how integrating STEM with other subjects can revolutionize K-12 education & prepare students for the future.
Content provided by Project Lead The Way
School & District Management Webinar How Pensions Work: Why It Matters for K-12 Education
Panelists explain the fundamentals of teacher pension finances — how they are paid for, what drives their costs, and their impact on K-12 education.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Personalized Learning Q&A Google Executive: AI Could 'Transform' School Into a 'Personal Learning Experience'
Google for Education's Shantanu Sinha discusses recent advances in digital technologies.
4 min read
Image of girl working with teacher on tablet computer.
iStock / Getty Images Plus
Personalized Learning 5 Essential Ingredients for Effective Tutoring
Experts outline what it takes to put in place tutoring programs that improve academic achievement.
2 min read
Young woman learning online
Personalized Learning Spotlight Spotlight on Personalized Learning
This Spotlight will help you learn effective strategies for tailoring instruction to individual students and more.

Personalized Learning Schools Are Spending Big Bucks on Online Tutoring. Here’s What They’ve Learned
Students need more than "homework help" to recover from pandemic learning loss. Is online tutoring up to the task?
8 min read
Young girl working at home with her teacher or tutor on the laptop in front of her.
iStock/Getty Images Plus