Find your next job fast at the Jan. 28 Virtual Career Fair. Register now.
Education Opinion

Results from the 2017 Khan Academy Study

By Justin Reich — July 17, 2014 3 min read

Follow on the heels of a two-year observational study by SRI, the U.S. Department of Education recently announced a three-year randomized control trial study of Khan Academy to be conducted by WestEd. I wanted to document my predictions now, and then three years hence we can look back through our Google Glass in our self-driving cars and see if I’m right. Or completely wrong. Or replaced by a robot.

WestEd’s basic research plan is to find a series of community colleges that are teaching Algebra I and not currently using Khan Academy, and then in the 2015-2016 school year randomly assign half of them to start using Khan. I predict the results will hew closely to the findings from the RAND study of Carnegie Learning’s Cognitive Tutor: Algebra. In 9th grade Algebra courses, the Rand study found no impact in year 1, and some positive impact in year two. A student in a Cognitive Tutor classroom scored about .2 standard deviations better than a student from a classroom without the tool. The study authors characterize the impact as follows, “The effect size of approximately 0.20 is educationally meaningful - equivalent to moving an Algebra I student from the 50th to the 58th percentile.”

If WestEd goes ahead with their plan for a one-year study, I would expect, on average, that the classrooms with Khan Academy would not look much different from those without Khan Academy. If they extended their study for a second year, I expect that they would find modest positive effects. I suspect that Khan Academy will provider greater benefit to students who come from more affluent families (with greater resources to take advantage of Khan Academy outside of school) and to the stronger students in the course. I base the latter point on some of the intuitions of teachers reported in the SRI study of Khan Academy, who articulated that they thought Khan Academy was more useful for stronger students.

There isn’t really a reason to believe that Khan Academy would work much better than any other intelligent tutoring system out there, and most of these systems have effects in randomized trials that range from no effects to fairly modest effects (we can debate how big a .2 standard deviation gain is in this context, but it’s not the dawn of a new paradigm in mathematics instruction). Of course there will be variation from site to site. In some places, Khan Academy will harm learning, and in some places the improvements will be stronger.

Here’s what I don’t think we will see. I think there will be no evidence that the use of Khan Academy in classes leads to greater opportunities for projects, challenging problems, peer-teaching, differentiated instruction, math talk, or deeper thinking.

In the schools where these things happen, educators focus on them. The logic of blended learning is something of a Rube Goldberg contraption: if you want rich project-based learning, then you should spend a bunch of your time, money, procurement energy, political will, and professional development resources on intelligent tutoring software. The software will make you more efficient in the classroom so that you finally free up the time that you needed for project-based learning (or math talk, or rich challenges, or peer learning, or whatever). It’s kind of a strange logic. You want more meaningful student-teacher interactions? OK, step 1, sit your kids in front of a randomized worksheet problem generator.

The schools that want more rich project-based learning, they focus on--wait for it--rich project-based learning. They put their time, money, procurement, political will, and professional development resources towards the desired activity. Sticking kids in front of computer-generated worksheets helps kids do better on other computer-generated worksheets.

I also don’t think students will like it all that much. As Jal Mehta wrote this week over at the Learning Deeply blog,

But what I can tell you from visits to blended classrooms and schools, in both traditional public and charter schools, is that students tend to find what exists thus far as fairly dull, lacking both the community and the accountability that comes with good face to face learning. A number of students told us at one highly celebrated blended school that they liked everything about the school except for the online learning!

I’m nearly certain that the WestEd Khan Academy study will reveal no breakthroughs, no disruptions to the order of teaching, no new paradigm that no educator--only a true innovator from outside the sector--could have thought of, and no revolutions that convince us the time has come to abandon our Prussian-inherited educational systems and pick up for the promised land. I bet teachers who try to use Khan Academy in community colleges and stick with it for a couple of years have classes that are a little bit better and students (particularly the more affluent or stronger students) who do a little better on math tests.

But, we’ll strap on our jetpacks in 2017 and find out!

The opinions expressed in EdTech Researcher are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Branding Matters. Learn From the Pros Why and How
Learn directly from the pros why K-12 branding and marketing matters, and how to do it effectively.
Content provided by EdWeek Top School Jobs
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
How to Make Learning More Interactive From Anywhere
Join experts from Samsung and Boxlight to learn how to make learning more interactive from anywhere.
Content provided by Samsung
Teaching Live Online Discussion A Seat at the Table With Education Week: How Educators Can Respond to a Post-Truth Era
How do educators break through the noise of disinformation to teach lessons grounded in objective truth? Join to find out.

EdWeek Top School Jobs

Human Resources Manager
Madison, Wisconsin
One City Schools
Elementary Teacher - Scholars Academy
Madison, Wisconsin
One City Schools
Special Education Teacher
Chicago, Illinois
JCFS Chicago
Communications Officer
Chattanooga, Tennessee
Hamilton County Department of Education

Read Next

Education Briefly Stated Briefly Stated: January 13, 2021
Here's a look at some recent Education Week articles you may have missed.
8 min read
Education Obituary In Memory of Michele Molnar, EdWeek Market Brief Writer and Editor
EdWeek Market Brief Associate Editor Michele Molnar, who was instrumental in launching the publication, succumbed to cancer.
5 min read
Education Briefly Stated Briefly Stated: December 9, 2020
Here's a look at some recent Education Week articles you may have missed.
8 min read
Education Briefly Stated Briefly Stated: Stories You May Have Missed
A collection of articles from the previous week that you may have missed.
8 min read