Meeting District Needs

Districts Struggle to Judge Ed-Tech Pilot Projects

By Sean Cavanagh — December 01, 2015 7 min read
  • Save to favorites
  • Print

Many school leaders see pilot tests of educational technology products as important opportunities to try out digital tools before their districts leap ahead with costly purchases.

But a new study suggests that school officials and tech developers often fail to set clear standards for judging the success of those trial runs, and that there isn’t an effective process for gathering feedback from teachers and students.

And even if products perform impressively during pilots, district budget cycles tend to prevent administrators from buying the tech tools they’re trying out anytime soon.

Those were among the findings of an analysis put forward by Digital Promise, a congressionally authorized nonprofit that has sought to shine a light on how districts buy educational technology and why the process seems to frustrate and confuse both companies and educators.

Districts typically agree to allow companies to test products in their classrooms in the hope that the technologies will help meet specific educational needs or forge new paths for teaching and learning. Ed-tech companies, in turn, use pilots to showcase their goods, refine them, and nurture relationships with potential clients who could buy their products for use across an entire district or set of schools.

Yet the Digital Promise study, which focused on the experiences of six districts around the country, lays bare the factors that can stymie both parties’ ambitions.

Many K-12 officials would like to be able to judge the value of ed-tech products by looking at their impact on student test scores—no surprise, given the pressure schools face to raise achievement. And some of the districts that participated in the study sought to track gains through test scores, too.

But the participating districts also said that when state assessment results were used to measure success, the results didn’t come in until after the academic year had ended, interfering with their decisions about how, or whether, to use the products the following school year. A few districts in the study sidestepped that problem by using locally crafted tests, which could be administered on their own schedules.

Yet the study’s authors warn of the limitations in trying to link test-score gains to the work of a single product, when the exam results could be swayed by other factors.

For an ed-tech developer, the study found, staging a successful pilot in a district in no way guarantees the company will land a contract to do work there. Districts’ tests of products for the study were held in the second semester and weren’t complete until around the end of the school year. That generally did not leave districts nearly enough time to set aside money and put in motion a procurement process to buy those goods for the next school year, even if that’s what K-12 leaders wanted.

“There’s an asynchronous calendar” between pilots and district purchases, said Valerie Adams-Bass, the lead author of the study, in an interview. And “many districts felt like, ‘We need to spend more time to try this product before we invest in it.’ ”

‘Mature’ Student Feedback

The research on pilots builds on a study commissioned last year by Digital Promise, an organization that seeks to improve education through technology and research, and by the Education Industry Association, as part of an analysis led by researchers at Johns Hopkins University.

That study found that companies have only a vague sense of districts’ buying needs and of how to interest them in their products. K-12 leaders, meanwhile, told researchers they were overwhelmed by product pitches and lacked the time and ability to evaluate them thoroughly.

The earlier study also revealed that districts rely heavily on pilots to test ed-tech products, but that their expectations for those trials can vary enormously.

To probe the topic in more depth, Digital Promise recruited six districts of varying sizes to participate in its new study: the District of Columbia; Fulton County, Ga.; Piedmont City, Ala.; South Fayette Township, Pa.; Vista (Calif.) Unified; and West Ada, Idaho. In some cases, district plans to pilot a product were already underway before the study began. Fulton County was the biggest district, with more than 95,000 students; Piedmont was the smallest, with about 1,200.

The districts tested a variety of products developed for different academic subjects and school needs. The District of Columbia schools used Newsela, a Web-based platform focused on literacy; Fulton County tested BrainPOP and IXL, producers of animated curricular content and of math and language arts content, respectively; Piedmont used Achieve 3000, in an effort to provide digital science and social studies content; South Fayette piloted Vex IQ, a robotics platform; Vista Unified piloted the Web-based math program ST Math; and West Ada tested ALEKS, another Web-based math program.

Districts officials who took part in the study said they valued teacher and student feedback about products but rarely collected it in a formal way, noted Adams-Bass, a postdoctoral fellow in education at the University of California, Davis, the institution that led the study. Information about pilot tests tended to trickle up from students to teachers and from teachers to principals, the study found.

The lack of a systematic way of capturing what students think of ed-tech products is a lost opportunity for developers, the study suggests.

Students’ comments about such products were “surprisingly mature, and they had particularly insightful comments about advice for education technology developers,” the authors found. “The student voice is vital to consider throughout a pilot process, as they are the true end-users.”

The insights offered by students helped shape the thinking of both district officials and developers working on the South Fayette Township district’s pilot, recalled Aileen Owens, the system’s director of technology and innovation. The district partnered with Carnegie Mellon University and the University of Pittsburgh on the pilot, along with Vex IQ, in trying to build students’ computational thinking and robotics programming. The district and its partners are making adjustments to how the product is being used this academic year, based on that what students liked, and what they didn’t, Owens said.

Winning Over Teachers

Among students’ initial observations from the South Fayette pilot: They wanted the platforms they used to challenge them more, and they wanted more real-world simulations, Owens recalled.

“That was eye-opening,” she said, “to learn about students’ discoveries and hear their voice.”

Rusty Greiff, the managing director of education ventures and a general partner at 1776, a Washington-based incubator and seed fund that supports ed-tech companies, agreed with the study’s authors on the importance of developers and K-12 officials working together to set clear goals for implementing pilots and evaluating products. He also agreed such coordination is often lacking.

Communication breaks down for several reasons, he said. Districts sometimes pilot digital products without thinking about whether the technology is aligned to their goals and objectives, and without making sure their teachers are on board, said Greiff, who has helped launch digital products and currently advises and serves on the boards of ed-tech companies.

“There are a lot of players and constituencies who have to be aligned for it to be effective,” Greiff said. “Pilots are hard to execute,” a fact that districts and companies “sometimes figure out quite late in the process.”

Companies and districts also overlook the amount of training and ongoing support educators need to make sure ed-tech products are used as intended during pilots, said Karen Billings, a vice president and the manager of the education technology industry network for the Software & Information Industry Association.

Assigning staff members to help teachers work through technical glitches and implement the product effectively “drives up costs for vendors,” Billings said. Yet while that training can bring “high costs for a company,” she said, “there’s a high reward.”

Tech developers also tend to overlook the day-to-day pressures teachers face—from managing a classroom to juggling myriad digital platforms—and how those factors keep pilots from occurring under ideal circumstances, he added.

Companies tend to think they’re going to “have pilots [occur on] their own island,” Greiff said. “Teachers don’t work on an island. They work in the classroom.”

A version of this article appeared in the December 09, 2015 edition of Education Week as Districts Struggle to Evaluate Pilot Projects for Ed Tech


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Classroom Technology Webinar
Transform Teaching and Learning with AI
Increase productivity and support innovative teaching with AI in the classroom.
Content provided by Promethean
Curriculum Webinar Computer Science Education Movement Gathers Momentum. How Should Schools React?
Discover how schools can expand opportunities for students to study computer science education.
School & District Management Webinar Fostering Student Well-Being with Programs That Work
Protecting student well-being has never been more important. Join this webinar to learn how to ensure your programs yield the best outcomes.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Meeting District Needs Districts' Back-to-School Shopping List: Masks, Gloves, Sanitizers and $25 Billion to Pay for It
When his staff’s tally for buying personal protective equipment reached $600,000, one superintendent said: “stop because we’re already bankrupt.”
4 min read
iStock/Getty Images Plus
Meeting District Needs Briefly Stated Briefly Stated: Stories You May Have Missed
A collection of stories you may have missed.
8 min read
Meeting District Needs Coronavirus Squeezes Supply of Chromebooks, iPads, and Other Digital Learning Devices
School districts are competing against each other for purchases of digital devices as remote learning expands to schools across the country.
7 min read
The New York City school system has been handing out laptops and other digital devices for students to use at home. Recently, it moved quickly to purchase 300,000 new iPads for remote learning.
The New York City school system has been handing out laptops and other digital devices for students to use at home. Recently, it moved quickly to purchase 300,000 new iPads for remote learning.
AP Photo/John Minchillo
Meeting District Needs Opinion Open Educational Resources Fill Gap for Underserved Students
The NAACP advocates the use of OER as a way to equalize learning resources at scale for all students, write Lisa Petrides and Barbara Dezmon.
Lisa Petrides & Barbara Dezmon
6 min read
Open Educational Resources Fill Gap for Underserved Students In the wake of NAACP’s endorsement of OER, states have a responsibility to address resource inequality, write Lisa Petrides and Barbara Dezmon