Opinion
Education Opinion

Computers Can Assess What Computers Do Best

By Justin Reich — April 29, 2014 6 min read
  • Save to favorites
  • Print

This is the third of three posts about blended learning, inspired by a recent Center for Education Policy Research conference at the Harvard Graduate School of Education that brought together researchers, software developers, funders, and educators to explore research frontiers in blended learning. The previous two posts, about educator frustration with data dashboards and excitement about embedding social-psychology based interventions in technology platforms.

In this post, I want to tackle what I think is the thorniest, most important problem within this system: the limitations of our current assessment technologies. The main problem is this: computers are mainly good at assessing the kinds of things we don’t need humans to do anymore. But let’s start at the beginning...

Assessment and Blended Learning

The basic model of personalized, blended learning as implemented in schools like Rocketship is as follows: For part of the school day, a kid sits at a computer. The computer teaches the kid stuff. The computer tests a kid on stuff. If the kid gets stuff right, she moves on to new topics. If not, the computer teaches the kid the same stuff, or sends data to the teacher about how to teach the kid that stuff (see previous post on how hard it is for computers to generate actionable data).

(As an aside, the places where this work happens sometimes look pretty funny. This picture from Rocketship looks like a call center that was decorated by the Easter Bunny.)

The lynchpin of this system is the computational assessment. The whole promise of computer-aided instruction is that each kid gets the stuff she needs, because the computer can quickly figure out what she knows and what she doesn’t. The kid can get fed more stuff more quickly because computers can assess students instantly and constantly; a teacher simply doesn’t have the time to figure out what stuff 30 elementary school students or 140 secondary school students need on a daily, hourly, minutely basis.

What Can Computers Assess?

So if computational assessment is the lynchpin of the system, then we need to ask: What human competencies can computers accurately assess? Basically they can do few things on their own, and one thing with some help. Multiple-choice questions, no problem. They can also evaluate quantitative questions with a single right answer that can be input with a keyboard (4, 2x+2, e^7, etc.). They can also evaluate computer code quite well: whether it works, how quickly it works, where it broke, whether it conforms to certain design specifications, etc. They are getting better at recognizing human speech and pronunciation, with some neat applications for language learning.

With some help, computers can also sort of evaluate human writing of about 400 words and up. They can’t really evaluate writing very well on their own, but if you take a sub-sample of at least a hundred essays and have humans grade them, then computers can predict the scores for the rest of the essays with about the same reliability as the humans who grade essays for standardized tests. (They don’t do as well with short documents, which don’t have enough data for algorithms to classify reliably.)

Overall, computers (without human training) are good at assessing the kinds of things--quantitative things, computational things--that computers are good at doing. Which is to say that they are good at assessing things that we no longer need humans to do anymore.

Take math, for instance. The Common Core State Standards for mathematical modeling describe six parts of mathematical modeling: 1) finding the problem, 2) representing the problem in equations, tables, or graphs, 3) calculating answers, 4) interpreting results, 5) validating conclusions, and 6) explaining reasoning.

In the real world, computers have a lock on #3 and humans have the edge in the other five. However, the only one of these that computers can reliably assess is #3 (unless we fudge and use multiple choice). Computers can’t tell us if a human has designed a good representation for a problem or provided a careful, reasoned defense of a mathematical approach to an issue.

If we specify an equation to solve, computers can check to see if a human has solved it correctly. That’s because the computer is good at solving that sort of thing. And the sorts of things computers are good at solving, we don’t really need humans to do any more.

Now, if a school’s main goal is to use software to program children to take tests like PARCC and Smarter Balance, this isn’t such a big deal. The PARCC and Smarter Balance tests will be constrained by these same dynamics, and they will be limited to the same type of computationally-assessable problems (except possibly some problems that might be farmed out to humans to grade). We can probably design software that programs children to take these tests. However, from a preparing humans for life in the real world, this is a real problem at the heart of blended learning. Software may prove best at preparing students for things that humans no longer need to do.

Future Directions in Computational Assessment Research

I’m quite interested in these assessment problems, especially for my work with HarvardX, and I’m interested in four lines of inquiry that might lead to better assessments in personalized online environments.

One of the fastest developing alternative to machine assessment is peer assessment. Can we get students to evaluate each other? Computers are very good at taking care of all the logistics so dozens or hundreds or thousands of students can evaluate each other’s work. And it may be that if you get five or six peers to evaluate another student, the average of their assessments might be pretty close to a teacher’s assessment. This might not be politically feasible in high-stakes situations (we don’t want little Johnny involved in high-stakes assessment of little Sally), but for various kinds of formative assessments, peer assessment tools might help us expand the range of the assessable.

Second, there are folks at HarvardX and elsewhere working on annotation tools; tools that allow learners to annotated and comment upon selections of text. Humanists have been using annotation and marginalia to develop and demonstrate their understanding for millennia, so this seems like a worthy exploration. If we ask students to explain their reasoning as they interpret a text, can we measure their ability to identify and comment upon relevant selections of text?

There are many different people interested in how game mechanics might be used in assessment situations. Virtual simulations can present people with complex, ill-structured problems, and then see how people approach these problems. In these assessment models, we might be able to evaluate both people’s solutions and the process a person follows to get to a solution. In many of these tools, however, the final step tends to be having students do some writing that looks an awful lot like a document-based question or other writing format that brings us back to all of the problems with assessing essays and other unstructured text.

As a final category, I think Sam Wineburg’s work with Beyond the Bubble is a worthy model of domain-specific, tractable, nuanced assessment questions. Beyond the Bubble asks students to perform specific applications of historical thinking skills with primary source documents. That model wasn’t developed with computational assessment in mind (and it would face challenges with the short length of student answers), but it would be interesting to try to get computer assessments to work in those domains.

This assessment question is really the lynchpin of blended learning models, or other models that depend upon software to assess and teach students. As long we face these limits on what computer tools can assess, we’ll face serious limits on the domains where computers can supplement or replace human teachers. The domains where computer assessment falls short may prove to be the most important domains for student learning.

For regular updates, follow me on Twitter at @bjfr and for my papers, presentations and so forth, visitEdTechResearcher.

The opinions expressed in EdTech Researcher are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.


Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
What’s Next for Teaching and Learning? Key Trends for the New School Year
The past 18 months changed the face of education forever, leaving teachers, students, and families to adapt to unprecedented challenges in teaching and learning. As we enter the third school year affected by the pandemic—and
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Webinar
How Data and Digital Curriculum Can Drive Personalized Instruction
As we return from an abnormal year, it’s an educator’s top priority to make sure the lessons learned under adversity positively impact students during the new school year. Digital curriculum has emerged from the pandemic
Content provided by Kiddom
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Judge's Temporary Order Allows Iowa Schools to Mandate Masks
A federal judge ordered the state to immediately halt enforcement of a law that prevents school boards from ordering masks to be worn.
4 min read
Iowa Gov. Kim Reynolds speaks to reporters following a news conference, Thursday, Aug. 19, 2021, in West Des Moines, Iowa. Reynolds lashed out at President Joe Biden Thursday after he ordered his education secretary to explore possible legal action against states that have blocked school mask mandates and other public health measures meant to protect students against COVID-19. Reynolds, a Republican, has signed a bill into law that prohibits school officials from requiring masks, raising concerns as delta variant virus cases climb across the state and schools resume classes soon. (AP Photo/Charlie Neibergall)
Education Hurricane Ida Deals New Blow to Louisiana Schools Struggling to Reopen
The opening of the school year offered teachers a chance to fully assess the pandemic's effects, only to have students forced out again.
8 min read
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021. Louisiana students, who were back in class after a year and a half of COVID-19 disruptions kept many of them at home, are now missing school again after Hurricane Ida. A quarter-million public school students statewide have no school to report to, though top educators are promising a return is, at most, weeks away, not months.
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021.
John Locher/AP
Education Massachusetts National Guard to Help With Busing Students to School
250 guard personnel will be available to serve as drivers of school transport vans, as districts nationwide struggle to hire enough drivers.
1 min read
Massachusetts National Guard soldiers help with logistics in this Friday, April 17, 2020 file photo, at a food distribution site outside City Hall, in Chelsea, Mass. Mass. Gov. Charlie Baker on Monday, Sept. 13, 2021, activated the state's National Guard to help with busing students to school as districts across the country struggle to hire enough drivers.
Massachusetts National Guard soldiers help with logistics in this Friday, April 17, 2020 file photo, at a food distribution site outside City Hall, in Chelsea, Mass.
Michael Dwyer/AP
Education FDA: ‘Very, Very Hopeful’ COVID Shots Will Be Ready for Younger Kids This Year
Dr. Peter Marks said he is hopeful that COVID-19 vaccinations for 5- to 11-year-olds will be underway by year’s end. Maybe sooner.
4 min read
Dr. Peter Marks, director of the Center for Biologics Evaluation and Research in the Food and Drug Administration, testifies during a Senate health, education, labor, and pensions hearing to examine an update from federal officials on efforts to combat COVID-19 on Capitol Hill in Washington on May 11, 2021. On Friday, Sept. 10, 2021, Marks urged parents to be patient, saying the agency will rapidly evaluate vaccines for 5- to 11-year-olds as soon as it gets the needed data.
Dr. Peter Marks, director of the Center for Biologics Evaluation and Research in the Food and Drug Administration, testifies during a Senate health, education, labor, and pensions hearing to examine an update from federal officials on efforts to combat COVID-19 on Capitol Hill in Washington on May 11, 2021.
Jim Lo Scalzo/AP