Beta testing is one of the most basic steps on the path to getting education products and ideas into the classroom—and, researchers and developers say, one of the trickiest to get right.
As applied to K-12, the term generally refers to the early stage of testing almost any product in schools—a game, an assessment, a software package, a personal computer—and then refining it, based on how well it works.
School districts typically agree to take part in beta testing because they will get access to a product or service they believe will help them—which could be a new learning tool for their students or a professional-development program for their staff members.
Yet beta testing can also pose challenges for both schools and product developers. When superintendents and school boards agree to beta-test a product, they know that they may be creating extra work for teachers and administrators who need to be trained in how to use it, and that their schools may be forced to carve out class time for trying it out.
Those concerns, among others, sometimes lead districts to reject offers to stage beta tests, to the frustration of developers desperate to take products for test flights in schools.
To complicate matters, beta testing can mean very different things within the research community, and within schools.
As in other fields, like medicine, education beta tests are often defined as efforts to test something through a series of pilots or experiments to see if a product does what it’s supposed to before it goes live on the market, said Grover J. “Russ” Whitehurst, a former director of the Institute of Education Sciences, the main research arm of the U.S. Department of Education.
In other cases, a beta test can include a far more structured process to evaluate something in a scientifically rigorous fashion, such as through a randomized control trial, said Mr. Whitehurst, who now directs the Brown Center, a research division at the Brookings Institution in Washington focused on school improvement. In those experiments, one group of students might be exposed to a product, while a second one is not, and results are compared between the two groups.
In any case, the process is “critical for any product that has any degree of innovation in it,” said Mr. Whitehurst, who conducted beta tests on reading and assessment products he helped develop before joining the federal government. “If the developer intends for it to accomplish something that hasn’t been accomplished before, it’s absolutely necessary,” he said.
Strengths and Shortcomings
Finding a district willing to act as a partner in beta testing is one thing; staging a successful pilot test that leads to a product breaking into the K-12 market is something else.
Elliot Soloway, a professor of computer science and engineering at the University of Michigan in Ann Arbor, who has conducted several beta tests as a product developer, has experienced that reality firsthand.
Several years ago, Mr. Soloway said, he worked with a company that was beta-testing pocket personal computers—meant to serve as “mobile learning environments"—among a relatively small group of teachers and about 150 students in a school district in an Eastern state. (He declined to name the school system or the state.) The teachers in the group quickly became adept at using the tool. Students’ test scores improved. So did their record in turning in homework on time.
“It went swimmingly,” Mr. Soloway recalled.
The trouble started when the product was scaled up for use by more than 50 teachers and 1,500 students. Costs rose. While the system’s software and hardware worked well in the original test group, unexpected problems emerged when teachers in the larger group used the devices, probably because those educators lacked the tech-savvy to resolve glitches on their own, Mr. Soloway speculated. Teachers confused by the technology tended to stop using the devices “dead in their tracks,” without providing developers with enough information to diagnose the problem, he said.
Another complication: While teachers in the original test group were able to craft learning activities to make use of the technology’s features, teachers in the larger group were unable to figure out how to make the technology work for their curricular needs—and believed doing so simply added to their workload, he added.
The pocket personal computer system never made it to the marketplace.
The project unraveled because of a computer “bug rebellion” and a “teacher rebellion,” Mr. Soloway said. “It was horrible. ... We got lured into thinking we had a successful pilot; we can roll it into the rest of the school. Not a chance.”
The most obvious temptation among developers is to structure the beta test in a way that will heighten the chances a product will succeed, which can undermine the process, Mr. Soloway said.
In those circumstances, “of course it’s going to succeed,” he said, but the developer finds “you don’t learn enough” about the product’s true strengths and shortcomings.
Mr. Soloway still consults with education companies, and he still does beta testing, but he says he has learned from earlier setbacks. Today, he says he pays more attention to providing teachers with tech support, and help crafting lessons that mesh with the technology. He adds that experience has taught him that developers and district officials need to work closely together to choose a range of teachers and students with different backgrounds, who are likely to have different degrees of willingness to accept the technology, rather than cherry-picking participants.
Michael Casserly, the executive director of the Washington-based Council of the Great City Schools, said that when district leaders resist allowing beta testing in their schools, it’s often because they worry that process isn’t transparent, and that the results may not be portrayed accurately. Concerns about creating new administrative burdens and training tend to be less important, he said.
The council will agree to help coordinate or recommend beta tests among its 67 member districts only if developers agree to make the results of those tests public—even if they produce undesirable results, Mr. Casserly said.
His group also expects developers to choose independent evaluators, rather than people predisposed to cast the results in a positive light. In addition, the council recommends that the individual districts it represents follow those same guidelines when considering allowing beta testing, independently of the organization.
When developers are told of the council’s expectations, “all of a sudden they’re not interested,” Mr. Casserly said. “The last thing they want is to test a product, then have all of the nation’s major school districts know that it doesn’t work.”
Evidence-Based Decisions
Developers sometimes conduct beta tests in different phases, with different goals in mind.
That was the case with an experiment being conducted by Melanie Stegman—the director of the learning technologies program at the Federation of American Scientists, in Washington—who is conducting a beta test of Immune Defense, a video game designed to teach students about immunology in an interactive and entertaining way.
Ms. Stegman is testing the game among students studying video-game design and other subjects at McKinley Technology High School, a public school in Washington. Those students are relatively adept with technology, but that’s OK, Stegman says—the goal of this part of the test is to understand on a basic level whether the game works without glitches, whether students understand how to play the game, and whether they enjoy it.
She said she also asks those teenagers for suggestions about its design.
Ms. Stegman is now recruiting teachers—some tech-savvy, some not—to participate in the next phase of her research, a randomized control trial which will ask them to use the game over three separate days, for one class period each day. The experiment will help determine whether students’ knowledge of cell biology increases as a result of playing the game, compared with other classroom activities considered to be best practices, she said.
At that point, “we’ll be able to test whether the game is really working,” she said. “Is it teaching kids biology?”
Chris Dasenbrook, a teacher who is overseeing the classes where Immune Defense is being tested, said the project appeals to him because it gives students insights on the challenges faced by actual game developers. He has carved out time for students to test the game in three different classes, including ones on interactive media and 2-D concepts.
“A lot of my kids play video games,” Mr. Dasenbrook said, but “it’s different when you’re pilot-testing. You’re chasing down bugs. You’re putting the program through its paces.”
Ultimately, he said, the beta test shows students the challenges in transforming an idea for a game into a “commercially viable product.”
Not all developers and researchers are able to find willing partners in K-12 to help them test their products.
When he was the director of the Institute of Education Sciences, Mr. Whitehurst said, members of the research and development community occasionally asked him to consider supporting a requirement that schools take part in beta testing in exchange for federal funding, such as Title I money—a step he considered a major overreach.
Still, Mr. Whitehurst said, federal officials should consider providing incentives to districts, possibly through grants provided for education research, to encourage beta testing.
Developers, he added, could make the process more attractive to districts by agreeing to carve out time during their beta tests to conduct research specific to the needs of those school systems and their personnel. Mr. Casserly of the Council of Great City Schools, meanwhile, believes districts and developers would benefit if school systems set clearer procedures for how beta tests are to be conducted and managed.
Encouraging more schools to take part in beta testing would have the side benefit of creating a new, more sophisticated class of consumers of educational products, Mr. Whitehurst added.
“It would be a way to get school leaders to buy into the importance of using evidence to make decisions,” he said.
