“Even if we answer optimistically, that is not a reason not to ask [these questions],’' he said.
Pamela Dubos, the chairman of the English department at nearby Bethel Park High School, spent a day here this summer observing this district’s new portfolio-assessment program.
Like a rapidly growing number of schools around the nation, Ms. Dubos said, hers is considering using portfolios as a way of measuring student performance in writing. She wanted an inside look at Pittsburgh’s program, which is further along than most.
“I’m impressed with the amount of resources the Pittsburgh school district put into it, as well as the amount of careful thought,’' she said. “That’s amazing. Some things are implemented in education without careful thought.’'
But Ms. Dubos, who came here during a week when more than two dozen teachers and supervisors were scoring a sample of the portfolios for the district, did not simply see how the process worked. She was part of the process itself, and her observations could help determine if portfolios become part of the assessment repertoire nationally.
Ms. Dubos was one of 20 educators and community officials who were overseeing the portfolio evaluation to ensure that it was done honestly and credibly.
If the “audit team’’ is satisfied with the process--and members expressed strong support for it--it will issue a report to the superintendent verifying the teachers’ results. The results will then be reported to the public as the first districtwide measure of writing performance using the new form of assessment.
The audit team was included in the process to test the notion that portfolios can serve as accountability measures as well as teaching tools, according to Paul G. LeMahieu, the director of the division of research, evaluation, and test development for the Pittsburgh school district.
“The system we have evolved over a four-year period of time, we are absolutely convinced, has a good effect on classroom practice,’' Mr. LeMahieu said. “The question is whether we can get evidence from it to make a good-faith accounting to the public.’'
“I am optimistic at this point,’' he added. “But we don’t have all the questions answered.’'
Dennie Palmer Wolf, a senior research associate at Harvard University’s graduate school of education, said Pittsburgh’s use of an audit system is a “good first step’’ to enable the district to use the new form of assessment as researchers here and elsewhere iron out the rough spots in the technology.
“This is not the end of the world,’' Ms. Wolf said. “What we’ve got here is, how to go on being accountable to the public as they figure out these things.’'
A Bold Step
Portfolio assessment, in which students are evaluated on the basis of work they complete as part of their classroom instruction, has rapidly emerged as a promising alternative to standardized tests as a way of measuring student achievement.
Vermont, for example, this year became the first state to use portfolios as part of its statewide assessment system. The state assessed 4th and 8th graders in mathematics and writing using portfolios, although several districts did not take part in the math assessment. <(See Education Week, May 20, 1992.)
And the National Assessment of Educational Progress in April released the results of a special study that tested the feasibility of conducting a national assessment of students’ in-class writing. (See Education Week, April 22, 1992.)
Here in Pittsburgh, the effort to use portfolios to gauge student performance began in 1987, when the district teamed up with researchers from Harvard and the Educational Testing Service to develop an unusual system for assessing performance in writing, visual arts, and music classrooms. Funded by the Rockefeller Foundation, the project, known as Arts PROPEL, allowed teachers to monitor student performance over the course of a year by evaluating students’ work and their reflections about the work. (See Education Week, Nov. 16, 1988.)
Since then, the project, now known simply as PROPEL, has spread to mathematics and science classrooms as well.
Roberta Camp, a senior researcher at the E.T.S., said teachers in Pittsburgh took a bold step by suggesting that the district use the portfolios to report student achievement in writing.
“Teachers originally thought portfolios would stand alongside [California Achievement Test] scores,’' Ms. Camp said. “But when CAT scores became the lingua franca of how schools were doing, a lot of language-arts teachers pulled grammar books out of the closet, and stopped asking kids to be reflective about their writing.’'
Typical, Not Exceptional
After pilot-testing the assessment in the 1990-91 school year, the district put it in place over the past year.
Under the program, students in grades 6-12 accumulate a folder of their writing over the course of a year. From the folder, they select a portfolio, which consists of pieces they consider “important,’' “satisfying,’' and “unsatisfying,’' along with any rough drafts and notes from conferences with teachers. They also fill out questionnaires indicating why they chose the pieces.
“I don’t know anybody who asks kids to select a piece that is unsatisfying, to put kids at risk that way,’' said Ms. Camp of the E.T.S. “I wouldn’t advise it, without the support Pittsburgh has generated in classrooms. It takes a lot of work with students for them to feel confident enough’’ to identify a piece as unsatisfying, she noted.
In addition to those works, the portfolio also includes a “free pick,’' or a writing sample that is intended to round out the portfolio, a “portrait of the student as a writer,’' and a final reflection on the student’s writing ability.
Mr. LeMahieu said the district avoided mandating the contents of the portfolios, and deliberately left vague the meaning of the terms “important,’' “satisfying,’' and “unsatisfying’’ so that students could be free to use their own judgments.
He also noted that, unlike other portfolio projects, such as Vermont’s, the Pittsburgh effort does not ask for students’ best work. Rather, he said, the portfolio assessment reflects what the community can typically expect from Pittsburgh students.
“This is not the point in education history when we should reassure the public that everything is O.K.,’' Mr. LeMahieu said. “We can’t talk exclusively about what happens when somebody gets lucky, and then have them go out and not be lucky.’'
High Standards
To create the reports on student writing performance, the district collected a random sample of 14 percent of the portfolios of 7th and 10th graders, the grades that will be reported to the public, and 8 percent to 10 percent of portfolios from students in the other grades.
The district then assembled a team of 25 teachers and three supervisors to score them. The group met here for a week immediately after the school year ended in June.
The scorers rated each portfolio, on a 1-to-6 scale, with a 6 representing outstanding performance, according to three dimensions: accomplishment in writing, the use of processes and strategies for writing, and development as a writer.
Each portfolio was read twice; if the scores on the two ratings diverged by more than 1 point, the portfolio was read a third time, independently, by an arbitrator.
Only 8.7 percent of the 1,700 portfolios scored this summer went to arbitration, Mr. LeMahieu noted, and many of those reflected divisions over whether students who presented no evidence of writing strategies should be scored as low performers.
The 6-point scale was designed so that there would be no middle point, according to JoAnn T. Eresh, director of the division of writing and speaking for the district.
In addition, she said, district officials insisted on keeping the three dimensions separate, and refused to compile them for a single score.
Students’ accomplishment in writing, she said, should be based on the product itself, not on the work that went into it.
“We admire Walt Whitman more because he wrote 15 editions of Leaves of Grass,’' Ms. Eresh pointed out.
She also noted that the standard for a score of 6 is high, and should reflect superior performance that all students should reach for. For high-school students, she said, a 6 should represent the top achievement of 12th graders in a gifted-students program.
But Arla Muha, an English teacher at South Vocational-Technical High School and one of the evaluators, questioned whether they should evaluate all the works according to a single standard.
“I have problems with comparing 9th-grade writing to a 12th-grade standard,’' she said. “I look at stuff that for a 10th grader is good. But it’s not a 5 or a 6. I’m having trouble with that.’'
Tough But Fair
Despite such concerns, teachers and audit-team members said that the assessment would yield an accurate picture of writing achievement in the Pittsburgh public schools.
“I am confident that the information in a portfolio, looked at with the care I saw, creates a much more accurate reflection of how young people are handling the challenge of incorporating thinking and writing than any other measure I have seen,’' said Joseph F. Dominic, a program officer for the Pittsburgh Foundation.
Ms. Eresh also observed that the portfolios provide sufficient information, including early drafts and el20lconference notes, to ensure that students are not plagiarizing or bringing in pieces that their parents or siblings wrote. One Pittsburgh high school recently was rocked with allegations of widespread cheating on tests and coursework. (See story, page 27.)
As a further check against cheating, said James F. Hertzog, the chief of the division of evaluation and reports for the Pennsylvania Department of Education and an audit-team member, the district can compare students’ portfolio scores with those from statewide writing tests.
Kathryn Howard, an 8th-grade teacher at Reizenstein Middle School, said she was confident that the scorers were tough and fair in their evaluations. But she added that she welcomed the auditors’ report as a way of convincing a skeptical public that that was the case.
“They would say, ‘Sure, Pittsburgh teachers grading Pittsburgh students give them high scores,’'' Ms. Howard said. “In fact, the opposite happened.’'
“But my saying that,’' she added, “is different from Bob Feir [the executive director of the Pennsylvania Board of Education] saying that or Joe Dominic saying that.’'
Not All News Is Good News
But the scorers and Mr. LeMahieu also suggested that the picture they present will be “an eye-opener.’'
In many respects, Ms. Camp said, the results represent an assessment of schools as much as of students.
“They may see that there is not as much process [writing] going on as they’d like to see,’' she said. “That’s information they need to know.’'
In part, whatever shortcomings show up in the results reflect the newness of the project, said Carolyn Olasewere, an English teacher at Westinghouse High School.
“Kids were responding to things they have never responded to,’' she said. “People don’t reflect. They are not shown enough things to know what’s wrong. You can’t expect them to say more than, ‘I didn’t capitalize.’''
In addition, Ms. Muha of South Vocational-Technical High School said, the large class sizes in the district make it difficult for teachers to work with students as closely as they would like to improve their writing.
“I wish I could with every paper get a chance to talk to every kid,’' she said. “I don’t have that time.’'
Ms. Muha argued that Pittsburgh’s system of dividing courses into semesters also inhibits writing instruction. The system was designed to help reduce the failure rate by allowing students a second chance to pass a course, she said, but in practice it is “atrocious.’'
“If you have the same kids [throughout the year], by February, they know each other, and have developed a rapport with you and with each other,’' Ms. Muha said. “Then, if you get a new class in, you have to build that rapport all over again.’'
But the vocational-school teacher predicted that writing performance would improve as teachers and students become more familiar with it, and cited her own experience as evidence.
“I could never get a whole class of auto-body boys to write poems,’' she said. “Now, they do it.’'
Asking More Questions
Perhaps equally important, Mr. LeMahieu said, the project could demonstrate to a national audience the feasibility of conducting a large-scale portfolio assessment.
“The simple fact that it worked, it’s safe to say, ‘Yes, it can work,’'' he said. “If it can work this well in a situation where money is as tight as here, I don’t see how it can’t work.’'
Nevertheless, he pointed out, district officials are conducting additional research to determine how well the system is working.
For example, Mr. LeMahieu said, the district has collected data on the time of day that evaluators read portfolios, to see if scores were different in the morning or the afternoon, and on their race and gender, to uncover any evidence of bias in their evaluations.
“Even if we answer optimistically, that is not a reason not to ask [these questions],’' he said.
He also said researchers are examining the effects on classroom practice of reporting the portfolio results publicly.
Ms. Wolf, the Harvard researcher, said such investigations are essential if portfolio assessment is to become part of the nation’s education system.
“Everybody in the world does portfolios,’' she said. “There is a kind of halo around them, as if the technology will solve the problems endemic to the system.’'
“But for portfolios to be even a part of school reform,’' Ms. Wolf added, “takes a long developmental process.’'
