Auditors Help Pittsburgh Make Sure Its Portfolio Assessment Measures Up

By Robert Rothman — August 05, 1992 11 min read
  • Save to favorites
  • Print

“Even if we answer optimistically, that is not a reason not to ask [these questions],’' he said.

Pamela Dubos, the chairman of the English department at nearby Bethel Park High School, spent a day here this summer observing this district’s new portfolio-assessment program.

Like a rapidly growing number of schools around the nation, Ms. Dubos said, hers is considering using portfolios as a way of measuring student performance in writing. She wanted an inside look at Pittsburgh’s program, which is further along than most.

“I’m impressed with the amount of resources the Pittsburgh school district put into it, as well as the amount of careful thought,’' she said. “That’s amazing. Some things are implemented in education without careful thought.’'

But Ms. Dubos, who came here during a week when more than two dozen teachers and supervisors were scoring a sample of the portfolios for the district, did not simply see how the process worked. She was part of the process itself, and her observations could help determine if portfolios become part of the assessment repertoire nationally.

Ms. Dubos was one of 20 educators and community officials who were overseeing the portfolio evaluation to ensure that it was done honestly and credibly.

If the “audit team’’ is satisfied with the process--and members expressed strong support for it--it will issue a report to the superintendent verifying the teachers’ results. The results will then be reported to the public as the first districtwide measure of writing performance using the new form of assessment.

The audit team was included in the process to test the notion that portfolios can serve as accountability measures as well as teaching tools, according to Paul G. LeMahieu, the director of the division of research, evaluation, and test development for the Pittsburgh school district.

“The system we have evolved over a four-year period of time, we are absolutely convinced, has a good effect on classroom practice,’' Mr. LeMahieu said. “The question is whether we can get evidence from it to make a good-faith accounting to the public.’'

“I am optimistic at this point,’' he added. “But we don’t have all the questions answered.’'

Dennie Palmer Wolf, a senior research associate at Harvard University’s graduate school of education, said Pittsburgh’s use of an audit system is a “good first step’’ to enable the district to use the new form of assessment as researchers here and elsewhere iron out the rough spots in the technology.

“This is not the end of the world,’' Ms. Wolf said. “What we’ve got here is, how to go on being accountable to the public as they figure out these things.’'

A Bold Step

Portfolio assessment, in which students are evaluated on the basis of work they complete as part of their classroom instruction, has rapidly emerged as a promising alternative to standardized tests as a way of measuring student achievement.

Vermont, for example, this year became the first state to use portfolios as part of its statewide assessment system. The state assessed 4th and 8th graders in mathematics and writing using portfolios, although several districts did not take part in the math assessment. <(See Education Week, May 20, 1992.)

And the National Assessment of Educational Progress in April released the results of a special study that tested the feasibility of conducting a national assessment of students’ in-class writing. (See Education Week, April 22, 1992.)

Here in Pittsburgh, the effort to use portfolios to gauge student performance began in 1987, when the district teamed up with researchers from Harvard and the Educational Testing Service to develop an unusual system for assessing performance in writing, visual arts, and music classrooms. Funded by the Rockefeller Foundation, the project, known as Arts PROPEL, allowed teachers to monitor student performance over the course of a year by evaluating students’ work and their reflections about the work. (See Education Week, Nov. 16, 1988.)

Since then, the project, now known simply as PROPEL, has spread to mathematics and science classrooms as well.

Roberta Camp, a senior researcher at the E.T.S., said teachers in Pittsburgh took a bold step by suggesting that the district use the portfolios to report student achievement in writing.

“Teachers originally thought portfolios would stand alongside [California Achievement Test] scores,’' Ms. Camp said. “But when CAT scores became the lingua franca of how schools were doing, a lot of language-arts teachers pulled grammar books out of the closet, and stopped asking kids to be reflective about their writing.’'

Typical, Not Exceptional

After pilot-testing the assessment in the 1990-91 school year, the district put it in place over the past year.

Under the program, students in grades 6-12 accumulate a folder of their writing over the course of a year. From the folder, they select a portfolio, which consists of pieces they consider “important,’' “satisfying,’' and “unsatisfying,’' along with any rough drafts and notes from conferences with teachers. They also fill out questionnaires indicating why they chose the pieces.

“I don’t know anybody who asks kids to select a piece that is unsatisfying, to put kids at risk that way,’' said Ms. Camp of the E.T.S. “I wouldn’t advise it, without the support Pittsburgh has generated in classrooms. It takes a lot of work with students for them to feel confident enough’’ to identify a piece as unsatisfying, she noted.

In addition to those works, the portfolio also includes a “free pick,’' or a writing sample that is intended to round out the portfolio, a “portrait of the student as a writer,’' and a final reflection on the student’s writing ability.

Mr. LeMahieu said the district avoided mandating the contents of the portfolios, and deliberately left vague the meaning of the terms “important,’' “satisfying,’' and “unsatisfying’’ so that students could be free to use their own judgments.

He also noted that, unlike other portfolio projects, such as Vermont’s, the Pittsburgh effort does not ask for students’ best work. Rather, he said, the portfolio assessment reflects what the community can typically expect from Pittsburgh students.

“This is not the point in education history when we should reassure the public that everything is O.K.,’' Mr. LeMahieu said. “We can’t talk exclusively about what happens when somebody gets lucky, and then have them go out and not be lucky.’'

High Standards

To create the reports on student writing performance, the district collected a random sample of 14 percent of the portfolios of 7th and 10th graders, the grades that will be reported to the public, and 8 percent to 10 percent of portfolios from students in the other grades.

The district then assembled a team of 25 teachers and three supervisors to score them. The group met here for a week immediately after the school year ended in June.

The scorers rated each portfolio, on a 1-to-6 scale, with a 6 representing outstanding performance, according to three dimensions: accomplishment in writing, the use of processes and strategies for writing, and development as a writer.

Each portfolio was read twice; if the scores on the two ratings diverged by more than 1 point, the portfolio was read a third time, independently, by an arbitrator.

Only 8.7 percent of the 1,700 portfolios scored this summer went to arbitration, Mr. LeMahieu noted, and many of those reflected divisions over whether students who presented no evidence of writing strategies should be scored as low performers.

The 6-point scale was designed so that there would be no middle point, according to JoAnn T. Eresh, director of the division of writing and speaking for the district.

In addition, she said, district officials insisted on keeping the three dimensions separate, and refused to compile them for a single score.

Students’ accomplishment in writing, she said, should be based on the product itself, not on the work that went into it.

“We admire Walt Whitman more because he wrote 15 editions of Leaves of Grass,’' Ms. Eresh pointed out.

She also noted that the standard for a score of 6 is high, and should reflect superior performance that all students should reach for. For high-school students, she said, a 6 should represent the top achievement of 12th graders in a gifted-students program.

But Arla Muha, an English teacher at South Vocational-Technical High School and one of the evaluators, questioned whether they should evaluate all the works according to a single standard.

“I have problems with comparing 9th-grade writing to a 12th-grade standard,’' she said. “I look at stuff that for a 10th grader is good. But it’s not a 5 or a 6. I’m having trouble with that.’'

Tough But Fair

Despite such concerns, teachers and audit-team members said that the assessment would yield an accurate picture of writing achievement in the Pittsburgh public schools.

“I am confident that the information in a portfolio, looked at with the care I saw, creates a much more accurate reflection of how young people are handling the challenge of incorporating thinking and writing than any other measure I have seen,’' said Joseph F. Dominic, a program officer for the Pittsburgh Foundation.

Ms. Eresh also observed that the portfolios provide sufficient information, including early drafts and el20lconference notes, to ensure that students are not plagiarizing or bringing in pieces that their parents or siblings wrote. One Pittsburgh high school recently was rocked with allegations of widespread cheating on tests and coursework. (See story, page 27.)

As a further check against cheating, said James F. Hertzog, the chief of the division of evaluation and reports for the Pennsylvania Department of Education and an audit-team member, the district can compare students’ portfolio scores with those from statewide writing tests.

Kathryn Howard, an 8th-grade teacher at Reizenstein Middle School, said she was confident that the scorers were tough and fair in their evaluations. But she added that she welcomed the auditors’ report as a way of convincing a skeptical public that that was the case.

“They would say, ‘Sure, Pittsburgh teachers grading Pittsburgh students give them high scores,’'' Ms. Howard said. “In fact, the opposite happened.’'

“But my saying that,’' she added, “is different from Bob Feir [the executive director of the Pennsylvania Board of Education] saying that or Joe Dominic saying that.’'

Not All News Is Good News

But the scorers and Mr. LeMahieu also suggested that the picture they present will be “an eye-opener.’'

In many respects, Ms. Camp said, the results represent an assessment of schools as much as of students.

“They may see that there is not as much process [writing] going on as they’d like to see,’' she said. “That’s information they need to know.’'

In part, whatever shortcomings show up in the results reflect the newness of the project, said Carolyn Olasewere, an English teacher at Westinghouse High School.

“Kids were responding to things they have never responded to,’' she said. “People don’t reflect. They are not shown enough things to know what’s wrong. You can’t expect them to say more than, ‘I didn’t capitalize.’''

In addition, Ms. Muha of South Vocational-Technical High School said, the large class sizes in the district make it difficult for teachers to work with students as closely as they would like to improve their writing.

“I wish I could with every paper get a chance to talk to every kid,’' she said. “I don’t have that time.’'

Ms. Muha argued that Pittsburgh’s system of dividing courses into semesters also inhibits writing instruction. The system was designed to help reduce the failure rate by allowing students a second chance to pass a course, she said, but in practice it is “atrocious.’'

“If you have the same kids [throughout the year], by February, they know each other, and have developed a rapport with you and with each other,’' Ms. Muha said. “Then, if you get a new class in, you have to build that rapport all over again.’'

But the vocational-school teacher predicted that writing performance would improve as teachers and students become more familiar with it, and cited her own experience as evidence.

“I could never get a whole class of auto-body boys to write poems,’' she said. “Now, they do it.’'

Asking More Questions

Perhaps equally important, Mr. LeMahieu said, the project could demonstrate to a national audience the feasibility of conducting a large-scale portfolio assessment.

“The simple fact that it worked, it’s safe to say, ‘Yes, it can work,’'' he said. “If it can work this well in a situation where money is as tight as here, I don’t see how it can’t work.’'

Nevertheless, he pointed out, district officials are conducting additional research to determine how well the system is working.

For example, Mr. LeMahieu said, the district has collected data on the time of day that evaluators read portfolios, to see if scores were different in the morning or the afternoon, and on their race and gender, to uncover any evidence of bias in their evaluations.

“Even if we answer optimistically, that is not a reason not to ask [these questions],’' he said.

He also said researchers are examining the effects on classroom practice of reporting the portfolio results publicly.

Ms. Wolf, the Harvard researcher, said such investigations are essential if portfolio assessment is to become part of the nation’s education system.

“Everybody in the world does portfolios,’' she said. “There is a kind of halo around them, as if the technology will solve the problems endemic to the system.’'

“But for portfolios to be even a part of school reform,’' Ms. Wolf added, “takes a long developmental process.’'

Related Tags:

A version of this article appeared in the August 05, 1992 edition of Education Week as Auditors Help Pittsburgh Make Sure Its Portfolio Assessment Measures Up


English-Language Learners Webinar Helping English-Learners Through Improved Parent Outreach: Strategies That Work
Communicating with families is key to helping students thrive – and that’s become even more apparent during a pandemic that’s upended student well-being and forced constant logistical changes in schools. Educators should pay particular attention
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Mathematics Webinar
Addressing Unfinished Learning in Math: Providing Tutoring at Scale
Most states as well as the federal government have landed on tutoring as a key strategy to address unfinished learning from the pandemic. Take math, for example. Studies have found that students lost more ground
Content provided by Yup Math Tutoring
Classroom Technology Webinar Building Better Blended Learning in K-12 Schools
The pandemic and the increasing use of technology in K-12 education it prompted has added renewed energy to the blended learning movement as most students are now learning in school buildings (and will likely continue

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Nearly a Million Kids Vaccinated in Week 1, White House Says
Experts say there are signs that it will be difficult to sustain the initial momentum.
4 min read
Leo Hahn, 11, gets the first shot of the Pfizer COVID-19 vaccine, Tuesday, Nov. 9, 2021, at the University of Washington Medical Center in Seattle. Last week, U.S. health officials gave the final signoff to Pfizer's kid-size COVID-19 shot, a milestone that opened a major expansion of the nation's vaccination campaign to children as young as 5. (AP Photo/Ted S. Warren)
Education How Schools Are Getting Kids the COVID Shot, and Why Some Aren’t
Some district leaders say offering vaccine clinics, with the involvement of trusted school staff, is key to helping overcome hesitancy.
5 min read
A girl walks outside of a mobile vaccine unit after getting the first dose of her COVID-19 vaccine, outside P.S. 277, Friday, Nov. 5, 2021, in the Bronx borough of New York. (AP Photo/Eduardo Munoz Alvarez)
Education Biden Administration Urges Schools to Provide COVID-19 Shots, Information for Kids
The Biden administration is encouraging local school districts to host vaccine clinics for kids and information on benefits of the shots.
2 min read
President Joe Biden, and first lady Jill Biden walk to board Marine One on the South Lawn of the White House in Washington, Saturday, Nov. 6, 2021. Biden is spending the weekend at his home in Rehoboth Beach, Del. (AP Photo/Alex Brandon)
Education Civil Rights Groups Sue Tennessee Over Law Against Transgender Student Athletes
The state law bars transgender athletes from playing public high school or middle school sports aligned with their gender identity.
3 min read
Amy Allen, the mother of an 8th grade transgender son, speaks after a Human Rights Campaign round table discussion on anti-transgender laws in Nashville, Tenn. on May 21, 2021.
Amy Allen, the mother of an 8th grade transgender son, speaks after a Human Rights Campaign round table discussion on anti-transgender laws in Nashville, Tenn. on May 21, 2021.
Mark Humphrey/AP