Teachers, Developers, Researchers Quickly Evaluate Ed Tech in N.Y.C.

By Benjamin Herold — February 04, 2015 5 min read
  • Save to favorites
  • Print

New York City schools are bringing together classroom teachers, software and app developers, and university researchers to conduct rapid-fire evaluations of new education technology products.

The project, known as the “Short Cycle Evaluation Challenge,” is part of an emerging national push to get teachers more involved in the development of ed-tech tools—and to provide better information to the school purchasing officials who buy those tools.

“It’s about using a process to evaluate who the products work for, when, and under what circumstances,” said Kara Chesal, the director of the 1.1 million-student district’s InnovateNYC Schools office, commonly referred to at the iZone.

Here’s how the concept looks in practice:

Working classroom teachers identify a problems they’d like technology to help address.

Officials from the iZone pair the teachers with companies offering products that might help.

Then researchers from Johns Hopkins University in Baltimore document how the tools are used in the classroom, conducting focus groups, administering surveys, and doing other analyses that ultimately lead to a report that is offered to both the developer and the district.

A major side benefit, participants in the challenge say, is the conversation and feedback that occurs between teachers and developers—two groups that typically operate in separate worlds.

The iZone’s $350,000 effort is one of three “test-bed networks” being funded via a $2 million worth of grants from the Bill & Melinda Gates Foundation. The others are in California’s Bay Area (the Learning Innovation Hub, launched in 2014 by the Silicon Valley Education Foundation, in cooperation with the NewSchools Venture Fund) and Chicago (LEAP Innovations.)

A ‘life-changing experience’

Five companies participated in the first round of New York’s short-cycle evaluation challenge last fall: two providers of educational math software (BuzzMath and Motion Math), a digital-reading platform (LightSail), and two digital student-portfolio management tools (Edusight and Open School ePortfolio.)

“We were at a stage where we had a basic product, but were looking to build new features, and we weren’t sure what to prioritize,” said Garros Li, the founder and president of Edusight, a one-and-a-half year old ed-tech company with offices in Toronto and San Francisco.

“It was a life-changing experience,” he said of participating in the challenge.

Li’s background is in business. After graduating from the University of Western Ontario, he says he worked for two years as a management consultant, helping Fortune 500 companies analyze data.

He left to found Edusight along with three others, based on the premise that teachers need better tools to collect, store, and organize signs that their student are learning.

“The systems they were using hadn’t changed in over 10 years,” Li said. “That really bugged me.”

The company set about building “an intuitive grade book with a built-in student-learning portfolio” that would allow teachers to see both quantitative measures (grades, student test scores, etc.) and qualitative measures (student work samples, teacher notes, videos of students in class) in the same place.

But what Edusight was missing, it turned out, was a good understanding of the low-tech workflow that teachers already use to gather and analyze such information.

“They had no clue what goes on in the life of a teacher,” said 12-year veteran Maria Arcodia, now a kindergarten teacher at Brooklyn Arbor Elementary School, in the borough’s Williamsburg section.

In her kindergarten class, Arcodia said, she has typically kept a running pen-and-paper checklist of how each of her students were doing on everything from letter recognition to phonemic awareness, bringing home charts and highlighters each weekend to group students in different skill and reading levels.

Arcodia and four Brooklyn Arbor colleagues applied last summer to be part of the iZone project because they wanted to make that process of analyzing student data more efficient.

She and her colleagues were pleasantly surprised to see how responsive Li and his fellow developers were to teachers’ experiences and concerns.

Their dialogue resulted in Edusight focusing its efforts on building a new mobile app that allowed teachers to use iPads to take and store photos of student work, as well as to organize students’ information by specific skills and academic standards (and not just by the type of assessment the teacher had given).

Quickly, Edusight became a useful tool the Brooklyn Arbor teachers wanted to use.

“It was really great when parents came in to be able to actually show them, ‘This is how your child sorted shapes,’” Arcodia said.

And for the company, the feedback was invaluable.

“We got lucky,” Li said. “The concept of having dedicated teachers testing your product and giving you detailed feedback is great.”

Systemic impact?

The impact of the iZone effort on the New York district’s ed-tech procurement process is yet to be seen.

Chesal, the iZone director, said the office is still working to figure out how to best bring the reports that resulted from the challenge to the principals and other district decision makers in a format that is useful.

The usefulness of the quick-turnaround “test-bed” evaluation also remains to be determined.

While it’s important to get real-world feedback as quickly as possible, so developers can improve their products and purchasing decisions can be made, there are also limits to the types of information that be collected and analyzed on such a short time frame.

The iZone’s efforts to determine the products’ impact on student achievement, for example, are based on entirely on self-reports from teachers about how their students are doing.

Nevertheless, Chesal said she’s optimistic that the process will lead to meaningful insights about how effective, fast evaluations of ed-tech products are best done. She noted that the New York, Chicago, and California efforts are all taking slightly different approaches.

And Emily Dalton Smith, the senior program officer at the Gates Foundation who is supporting the three networks, said the work is still early, but promising.

“We have been really thrilled to see a lot of activity around making sure kids and teachers are really driving ed-tech product development,” she said.

Photo: Garros Li, center, founder and president of ed-tech company Edusight, listens to teachers from New York City’s Brooklyn Arbor Elementary School. -- Courtesy of Edusight and InnovateNYC Schools.

The Bill & Melinda Gates Foundation provides support for Education Week’s coverage college- and career-ready standards. Education Week retains sole editorial control over the content of this coverage.

See also:

for the latest news on ed-tech policies, practices, and trends.

A version of this news article first appeared in the Digital Education blog.

Commenting has been disabled on effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Ensuring Continuity of Learning: How to Prepare for the Next Disruption
Across the country, K-12 schools and districts are, again, considering how to ensure effective continuity of learning in the face of emerging COVID variants, politicized debates, and more. Learn from Alexandria City Public Schools superintendent
Content provided by Class
Teaching Profession Live Online Discussion What Have We Learned From Teachers During the Pandemic?
University of California, Santa Cruz, researcher Lora Bartlett and her colleagues spent months studying how the pandemic affected classroom teachers. We will discuss the takeaways from her research not only for teachers, but also for

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Curriculum He Taught About White Privilege and Got Fired. Now He's Fighting to Get His Job Back
Matthew Hawn is an early casualty in this year's fight over how teachers can discuss with students America's struggle with racism.
13 min read
Social studies teacher Matthew Hawn is accused of insubordination and repeated unprofessional conduct for sharing Kyla Jenèe Lacey's, 'White Privilege', poem with his Contemporary Issues class. Hawn sits on his couch inside his home on August 17, 2021.
Matthew Hawn is accused of insubordination and repeated unprofessional conduct for lessons and materials he used to teach about racism and white privilege in his Contemporary Issues class at Sullivan Central High School in Blountville, Tenn.<br/>
Caitlin Penna for Education Week
Curriculum What's the Best Way to Address Unfinished Learning? It's Not Remediation, Study Says
A new study suggests acceleration may be a promising strategy for addressing unfinished learning in math after a pandemic year.
5 min read
Female high school student running on the stairs leads to an opportunity to success
CreativaImages/iStock/Getty Images Plus
Curriculum School Halts Use of Fictional Book in Which Officer Kills a Black Child
Fifth graders in at least one Broward County school were assigned to read a book that critics say casts police officers as racist liars.
Rafael Olmeda, South Florida Sun-Sentinel
5 min read
Broward County School Board member Lori Alhadeff listens during a meeting of the Broward County School Board, Tuesday, March 5, 2019, in Fort Lauderdale, Fla.
Broward County School Board member Lori Alhadeff listens during a meeting of the Broward County School Board in Fort Lauderdale, Fla. Alhadeff told the South Florida Sun-Sentinel that she does not feel like the book "Ghost Boys" is appropriate for 5th graders.
Lynne Sladky/AP
Curriculum Opinion Introducing Primary Sources to Students
Five educators share strategies for introducing primary sources to students, including English-language learners.
12 min read
Images shows colorful speech bubbles that say "Q," "&," and "A."