School & District Management Opinion

To Ensure Success, First Define the Black Box of Innovation

By Beth Holland — February 17, 2017 3 min read
  • Save to favorites
  • Print

I first read about the notion of the “black box” in the context of the economics and education. Economic researchers frequently write about inputs and outputs. They examine what goes into the black box of schools in terms of time, money, or resources and then measure what comes out - student test scores, teacher evaluations, etc. However, economists rarely open that box.

Take the example of increased teacher salaries. An economic study may suggest that schools who receive more money to pay teachers (input) show increased student test scores (output). The causal relationship that higher paid teachers lead to greater student achievement may seem logical based on an inference such as higher quality teachers may receive a higher pay. But what actually happened inside the school? What did those teachers who received more money actually do to improve student performance? Can the improvement really be attributed to the teachers and their salaries, or did something else occur?

In their 2012 article, Theoretical Frameworks to Guide School Improvement, authors Evans, Thornton, and Usinger explain that most educational reform efforts occur absent a defined theory of change. Instead, schools and districts implement new plans or programs without defining why they believe that these change efforts should work; what they hope to achieve; and how they might measure success. In concrete terms, consider the technology programs, curriculum overhauls, and even STEM and Maker movements that continue to be implemented with a black box approach.

Schools roll out new devices and programs (inputs) but neither identify the outputs and outcomes that they hope to achieve nor the logic detailing the process for change. As a result, educators and administrators often lament that their new program failed to gain traction or abandon the efforts because they lack positive results. However, without first delineating the inner workings of the black box, it is virtually impossible to fully understand what happened inside. What actually failed: the theory or the implementation?

Though researchers Leviton and Lipsey wrote about the idea of “treatment theory” in reference to formal, empirical evaluation studies, consider the possibility if we applied their recommendations to programs in schools. First, before designing any sort of program, define the precise problem as it exists in context. If iPads or Chromebooks or Makerspaces are the desired input, what problem are they trying to solve? Next, establish exactly what should happen within the program. What activities will students complete and when? What professional development might teachers require and with what frequency? Then, with the process laid out, stipulate any factors that may impact your planning or mitigate the effects of your program. Contemplate everything from parental concerns to teacher buy-in to administrator support. Finally, describe the desired outcomes. What new knowledge, attitudes, beliefs, or skills should result from this new program?

It always saddens me when I hear that a school made an initial investment in a device, or initiative and then abandoned it because of “failure” -- especially when similar organizations have succeeded with the same endeavor. That prior success implies that the root of the problem may be the execution of the program and not the underlying premise. Imagine the possibilities for new innovations if teachers, administrators, students, and the broader community all understood the inner workings of the black box before beginning something new. Hopefully, it would ensure that the theory would support the implementation, and there would be nothing to blame.


Evans, L., Thornton, B., & Usinger, J. (2012). Theoretical frameworks to guide school improvement. NASSP Bulletin, 96(2), 154-171. doi:10.1177/0192636512444714

Leviton, L. C., & Lipsey, M. W. (2007). A big chapter about small theories: Theory as method: Small theories of treatments. New Directions for Evaluation, 27-62. doi:10.1002/ev.224

Related Tags:

The opinions expressed in EdTech Researcher are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.