Overwhelming Response

By Ann Bradley — September 01, 1995 5 min read

For nearly a decade, the National Board for Professional Teaching Standards has been laboring to create an assessment system that will identify outstanding teachers. Along the way, it has learned some expensive lessons. One of the most recent is that less can be more.

That, at least, is what an analysis of the board’s first round of teacher assessments has revealed. The evaluators concluded that the portfolios candidates for national certification compiled last year overwhelmed both the teachers and those who scored them. What’s more, the reams of commentary and explanation that some teachers churned out for the portfolios did not yield enough information to be worth that kind of effort.

“You only get a glimpse of candidates’ practice with these huge exercises,” says James Smith, senior vice president of the board, which was created in 1987 to set standards for accomplished teaching and certify those who meet them.

The package of assessments—which in January yielded the board’s first group of nationally certified teachers—included an exercise called “Teaching and Learning.” The exercise asked candidates to write a narrative about what went on in just one of their classes over a three-week period. They were also asked to include samples of students’ work and a videotape of the class in action. That one exercise took evaluators four hours per teacher to score and ended up capturing a relatively narrow slice of a teacher’s job.

Despite such shortcomings, a number of observers say the performance exercises the board has developed so far represent an enormous leap forward in teacher assessment. “There is no question in my mind that they are dramatically better than anything else out there,” says Lee Shulman, a professor of education and psychology at Stanford University who is helping the privately organized board rethink some aspects of the exercises.

Part of what makes the new assessments different is the view of teaching that undergirds them. Teachers are expected to be reflective about their practice and oriented toward helping their students think critically. But in asking teachers to write about their teaching, the assessment developers learned that reflection, however desirable, does not come easily to teachers. “They are not very good at it, they are not trained to do it, and they don’t do it normally,” Smith says. “They can do a pretty good job describing and telling you what happened, but most don’t do a good job analyzing.”

The board will try to address all of these problems as it works on the many other assessments currently under development. For example, instead of asking teachers to think in general terms about their practice, the board may ask a series of focused questions. High school math teachers might be told to produce a 20-minute, uninterrupted videotape of themselves teaching a large group of students to reason mathematically. Then they might be asked questions about what came before and after the lesson. Another exercise might question teachers about how they would work with small groups of students on a different type of math problem.

The point, Smith explains, would be to capture samples of different kinds of teaching in different classrooms that would paint a fuller picture of a teacher’s work. “It’s harvesting their practice,” he says, “rather than requiring them to do things they’re not used to doing.”

Mari Pearlman, an assessment developer at the Educational Testing Service who is working with the board, says she and others have learned that they need to be specific with candidates. “There was an original sense that if you were talking to professionals, you shouldn’t be directive,” she says. “One of the things we’ve learned is you have to be directive. Teachers say, `Just say exactly what it is I should produce here.’ “

That includes such things as setting page limits on written exercises, which the first assessments did not do. As a result, teachers wrote wildly varying amounts. More focused exercises should prove less overwhelming to teachers and their evaluators. Teachers who looked at the initial math exercises that ETS is preparing for the board thought they were wonderful, Pearlman says, but they could not imagine doing them. “That’s a bad thing to hear,” she adds. “You’re not satisfying your customers.”

According to Smith, videotapes and samples of students’ work provided more compelling evidence about teachers than their own written commentaries. One reason for that, he says, is the lack of corroboration for what teachers write about themselves. One teacher described herself as “student centered” and then on the videotape asked and answered all the questions during a discussion with students.

The interview component of the examination, conducted at assessment centers after the portfolios were completed, also proved to be more difficult than expected. Interviewers asked candidates about the videotapes they had made of particular lessons, but because it was difficult to find, train, and pay highly skilled interviewers, the sessions were not as revealing and cost-effective as other assessment methods.

Some experts have urged the national board to make classroom observation a key component of the teacher evaluations, but financial constraints make such a step unlikely. Another idea was to score candidates’ portfolios before signing them up for the assessment-center exercises. This two-step system would save the board money. Candidates who did poorly on the portfolio would skip the assessment center altogether.

Observers say one reason the system is currently so expensive—assessing the first group of teachers cost roughly $4,000 each, far more than the $975 fee candidates pay—is that it’s brand-new. “What you have at the moment is something very much like a space shuttle, in the sense that every system has a backup,” Shulman of Stanford says. “The question now is how far to reduce a lot of that redundancy and keep the assessment safe for everybody to use.”

Although some of the problems associated with the new assessments have led to severe cost overruns, members of the national board say there are reasons to be upbeat. They point out, for example, that many of the teachers who took part in last year’s field test had nothing but praise for the process. “They say how good they think it was, that it did tap into what they do every day in the classroom,” says Joyce Ojibway Jennings, a San Diego teacher who sits on the 63-member board. “For me, that was even more important than the psychometrics.”

Nevertheless, the board has decided to slow the development of a number of assessments and concentrate instead on retooling the two now being offered. Although those two—for generalist and English/language arts teachers who work with early adolescents—were offered this past year, neither will be during the 1995-96 school year. Both will be redesigned and then reintroduced in 1996-97.

“The main idea is to prune down these exercises,” Smith says, “so they require less work on the part of the candidate—less busywork—and are a lot more focused.”

A version of this article appeared in the September 01, 1995 edition of Teacher as Overwhelming Response