Follow on the heels of a two-year observational study by SRI, the U.S. Department of Education recently announced a three-year randomized control trial study of Khan Academy to be conducted by WestEd. I wanted to document my predictions now, and then three years hence we can look back through our Google Glass in our self-driving cars and see if I’m right. Or completely wrong. Or replaced by a robot.
WestEd’s basic research plan is to find a series of community colleges that are teaching Algebra I and not currently using Khan Academy, and then in the 2015-2016 school year randomly assign half of them to start using Khan. I predict the results will hew closely to the findings from the RAND study of Carnegie Learning’s Cognitive Tutor: Algebra. In 9th grade Algebra courses, the Rand study found no impact in year 1, and some positive impact in year two. A student in a Cognitive Tutor classroom scored about .2 standard deviations better than a student from a classroom without the tool. The study authors characterize the impact as follows, “The effect size of approximately 0.20 is educationally meaningful - equivalent to moving an Algebra I student from the 50th to the 58th percentile.”
If WestEd goes ahead with their plan for a one-year study, I would expect, on average, that the classrooms with Khan Academy would not look much different from those without Khan Academy. If they extended their study for a second year, I expect that they would find modest positive effects. I suspect that Khan Academy will provider greater benefit to students who come from more affluent families (with greater resources to take advantage of Khan Academy outside of school) and to the stronger students in the course. I base the latter point on some of the intuitions of teachers reported in the SRI study of Khan Academy, who articulated that they thought Khan Academy was more useful for stronger students.
There isn’t really a reason to believe that Khan Academy would work much better than any other intelligent tutoring system out there, and most of these systems have effects in randomized trials that range from no effects to fairly modest effects (we can debate how big a .2 standard deviation gain is in this context, but it’s not the dawn of a new paradigm in mathematics instruction). Of course there will be variation from site to site. In some places, Khan Academy will harm learning, and in some places the improvements will be stronger.
Here’s what I don’t think we will see. I think there will be no evidence that the use of Khan Academy in classes leads to greater opportunities for projects, challenging problems, peer-teaching, differentiated instruction, math talk, or deeper thinking.
In the schools where these things happen, educators focus on them. The logic of blended learning is something of a Rube Goldberg contraption: if you want rich project-based learning, then you should spend a bunch of your time, money, procurement energy, political will, and professional development resources on intelligent tutoring software. The software will make you more efficient in the classroom so that you finally free up the time that you needed for project-based learning (or math talk, or rich challenges, or peer learning, or whatever). It’s kind of a strange logic. You want more meaningful student-teacher interactions? OK, step 1, sit your kids in front of a randomized worksheet problem generator.
The schools that want more rich project-based learning, they focus on--wait for it--rich project-based learning. They put their time, money, procurement, political will, and professional development resources towards the desired activity. Sticking kids in front of computer-generated worksheets helps kids do better on other computer-generated worksheets.
I also don’t think students will like it all that much. As Jal Mehta wrote this week over at the Learning Deeply blog,
But what I can tell you from visits to blended classrooms and schools, in both traditional public and charter schools, is that students tend to find what exists thus far as fairly dull, lacking both the community and the accountability that comes with good face to face learning. A number of students told us at one highly celebrated blended school that they liked everything about the school except for the online learning!
I’m nearly certain that the WestEd Khan Academy study will reveal no breakthroughs, no disruptions to the order of teaching, no new paradigm that no educator--only a true innovator from outside the sector--could have thought of, and no revolutions that convince us the time has come to abandon our Prussian-inherited educational systems and pick up for the promised land. I bet teachers who try to use Khan Academy in community colleges and stick with it for a couple of years have classes that are a little bit better and students (particularly the more affluent or stronger students) who do a little better on math tests.
But, we’ll strap on our jetpacks in 2017 and find out!