Special Report
Meeting District Needs

‘Red Flags’ to Look for When Evaluating Personalized Learning Products

By Michele Molnar — October 18, 2016 6 min read
  • Save to favorites
  • Print

Personalized learning: Is it an educational imperative, a marketing strategy for an ed-tech product, or both?

Too often, teachers and administrators say, they find that personalized learning is used by companies as mere buzzwords to promote a run-of-the-mill digital tool.

“In the marketing literature, this term is overused,” said Devin Vodicka, the superintendent of the Vista, Calif., school district. “Many products that someone claims are personalized are actually just a series of digital worksheets.”

But educators are finding ways to sort the real personalized potential from the empty promises of some ed-tech products.

For instance, Vista uses its own Personal Learning Pathway framework to evaluate products, said Vodicka.

Some of the key questions educators in his district ask:

• Is this product based on a student profile?

• Is there an integrated technology component that includes two-way communication?

• Does the product include student choice and pathways?

• How does it fit with the learning environment, and more broadly, is it connected with real-world opportunities?

• Does it use a competency-based model, in which students move at their own pace as they master academic content?

Last month, Vista hosted 42 superintendents from across the country, who discussed that and other approaches as part of an ongoing cohort studying personalized learning through AASA, The Superintendent’s Association.

The most important starting point to vet a product, for Barton Dassinger, the principal of Chavez Elementary in Chicago, is this query: Does this product improve student learning better than the alternative to using it? And does the company generate reports in such a way to allow for an analysis that will answer that question?

The starting points of “what is our need?” and “what is our goal?” are important for Théa Williams, who is both a technology teacher for pre-K-5 and the technology coordinator for Brooklyn Arbor Elementary School in New York City. As an iZone pilot-team leader through the city’s department of education, she said she has learned to move beyond the catchphrases for products to identify the data needed.

Red flags about the “personalized learning” label abound, according to Amy Nowell, the director of research for LEAP Innovations, a Chicago-based nonprofit that works to build educational innovations by connecting schools and businesses. From her experience testing ed-tech products with evidence-based research methods, Nowell, and other experts have identified several red flags educators should watch for:

Questionable Student Agency

Buzzwords in the Marketplace

This is a fundamental element of personalization. Can the students take ownership of their own learning by setting their own goals? Can they track their own progress?

Nowell recommends that teachers “test drive” products as though they are students and make sure they understand what the data mean.

For instance, if students aim to learn a particular unit within a specified period of time, and the product provides feedback that “you have only completed one activity” in that time frame, what, she asked, does that mean to a student trying to complete and comprehend the full unit?

Inadequate Content

By definition, personalized learning allows students to move at their own pace through material.

“There’s never a classroom where every student is average,” said Nowell. “We’ve had a number of teachers who were really disappointed when they dug in, and two months into the year, their brighter kids are looking for material that wasn’t there.”

While ed-tech products can often be retrofitted to accommodate material for advanced or struggling students, that’s often a “clunky” solution, she said. A digital resource with “only 130 lessons” is unlikely to have enough content to go up or down two grade levels.

Useless Data

Products for personalized learning generally produce a lot of data. Williams recommends educators first ask: “How do I make sense of this data?”

In theory, experts say a data dashboard should help students and teachers understand what the metrics are, how a certain metric was arrived at, and what it means for student learning. But that is not always the case.

For example, one product reported to teachers the percent of total lessons each student completed in the 4th grade curriculum. “Johnny has completed 4 percent of the lessons has no actual meaning to anybody,” Nowell said. “It’s not tied to what students are learning. It’s not tied to learning standards or mastery of content. Johnny could have clicked through, ‘completed’ them, and gotten them all wrong.”

Lacking Recommendations

“I need the data to help make instructional decisions,” said Williams.

The problem is that automating the process of using data to inform educators’ decisionmaking is still largely an unmet need. “I don’t feel personalized ed-tech products have mastered that yet,” she said. “I’d like to see more automation, providing data to teachers to make instructional interventions in the learning path.”

Poorly Aligned Assessments

Generally, the embedded assessment questions in personalized-learning products have had no external validation, Nowell said.

When piloting their products, companies want to know what assessments students will take, whether it’s from one of the common-core-aligned testing consortia or another well-known test. But because educational technology generally doesn’t have the resources for rigorous assessment design and validation, the assessment data that are generated from the trials might not align with the tests that eventually will be given to students, she said.

Classroom-Integration Problems

A company should be able to inform teachers how to best integrate their products into the classroom structure. Does the product work best in classrooms where groups of students rotate from one station to another, while the teacher instructs another small group? Can students work on it together, or solo?

“We’ve had teachers who started with the station-rotation model but had to revert to the typical classroom model,” said Nowell, “and it was really about the way the technology was set up, not about the kids not being mature enough to work that way.”

Another question is, “Who assigns the next piece of content? The teacher or the software?” Teachers want both options, she said.

Little Evidence It Works

Nowell cautions teachers not to put too much stock in testimonials from other users of a product, five-star ratings on websites, or anecdotes about how a product is used. “Press companies for hard data about documented outcomes,” she advised.

“Ask them for any studies done on how well their products impact student outcomes,” she said. For instance, companies should be able to provide empirical data on product effectiveness as it relates to student achievement. “Even case studies, if done well and based on a similar context to your classroom, can provide a powerful indication of how a product could potentially work for you.”

Measures of student learning can include nationally standardized test scores, as well as more timely measures of student engagement and motivation for learning, she added.

Lacking the Personal Perspective

Personalized learning means taking each child’s uniqueness into account.

“There are all these factors you have to consider: culture, family background, interests, learning strengths, social-emotional development. We could go on and on,” said Williams, of Brooklyn Arbor in New York. “We’re talking about a human being.”

Teachers would like to see products that provide them with more in-depth insights into their students.

Williams said she sees educators’ excitement about the prospect of personalization. But personalized-learning experts generally agree that most ed-tech products are not geared toward students’ individual backgrounds and interests.

“When they do find products that really work for them, and respond to how students answer questions then make adjustments accordingly, teachers can be so much more effective,” she said. “Then they can do the things that the personalized learning tool can’t do and focus their time and energy on that.”

Coverage of the implementation of college- and career-ready standards and the use of personalized learning is supported in part by a grant from the Bill & Melinda Gates Foundation. Education Week retains sole editorial control over the content of this coverage.
A version of this article appeared in the October 19, 2016 edition of Education Week as 8 ‘Red Flags’ to Look for in Products


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Mathematics Webinar
Pave the Path to Excellence in Math
Empower your students' math journey with Sue O'Connell, author of “Math in Practice” and “Navigating Numeracy.”
Content provided by hand2mind
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Recruitment & Retention Webinar
Combatting Teacher Shortages: Strategies for Classroom Balance and Learning Success
Learn from leaders in education as they share insights and strategies to support teachers and students.
Content provided by DreamBox Learning
Classroom Technology K-12 Essentials Forum Reading Instruction and AI: New Strategies for the Big Education Challenges of Our Time
Join the conversation as experts in the field explore these instructional pain points and offer game-changing guidance for K-12 leaders and educators.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Meeting District Needs Districts' Back-to-School Shopping List: Masks, Gloves, Sanitizers and $25 Billion to Pay for It
When his staff’s tally for buying personal protective equipment reached $600,000, one superintendent said: “stop because we’re already bankrupt.”
4 min read
iStock/Getty Images Plus
Meeting District Needs Briefly Stated Briefly Stated: Stories You May Have Missed
A collection of stories you may have missed.
8 min read
Meeting District Needs Coronavirus Squeezes Supply of Chromebooks, iPads, and Other Digital Learning Devices
School districts are competing against each other for purchases of digital devices as remote learning expands to schools across the country.
7 min read
The New York City school system has been handing out laptops and other digital devices for students to use at home. Recently, it moved quickly to purchase 300,000 new iPads for remote learning.
The New York City school system has been handing out laptops and other digital devices for students to use at home. Recently, it moved quickly to purchase 300,000 new iPads for remote learning.
AP Photo/John Minchillo
Meeting District Needs Opinion Open Educational Resources Fill Gap for Underserved Students
The NAACP advocates the use of OER as a way to equalize learning resources at scale for all students, write Lisa Petrides and Barbara Dezmon.
Lisa Petrides & Barbara Dezmon
6 min read
Open Educational Resources Fill Gap for Underserved Students In the wake of NAACP’s endorsement of OER, states have a responsibility to address resource inequality, write Lisa Petrides and Barbara Dezmon