Education

RAND Study Finds Serious Problems in Vt. Portfolio Program

By Robert Rothman — December 16, 1992 6 min read
  • Save to favorites
  • Print

A report analyzing Vermont’s pioneering assessment system has found severe problems with it and raised serious questions about alternative forms of assessment.

The Vermont system, which is being closely watched by educators around the country, is the first statewide assessment program to measure student achievement in part on the basis of portfolios.

But the report by the RAND Corporation, expected to be released this week, found that the “rater reliability’’ in scoring the portfolios--the extent to which scorers agreed about the quality of a student’s work--was very low. The researchers urged the state to release the assessment results only at the state level.

Daniel M. Koretz, a senior social scientist at RAND and the report’s author, said the low levels of reliability indicate that the scores are essentially meaningless, since a different set of raters could come up with a completely different set of scores.

“If you’re not rating reliably, you’re not rating,’' he said. “You can’t measure anything unless you measure it reliably.’'

The report recommends a number of changes that could boost reliability, including improving the training of teachers who rate the portfolios. Mr. Koretz noted that Vermont is considering taking steps in those directions.

But he cautioned that reliability might continue to pose a problem in Vermont’s program, as well as in other types of performance assessments and portfolios, in which teachers rate diverse student work.

Advocates of alternative assessment need to gather data on the quality of the new methods before using them to replace traditional tests, Mr. Koretz said.

“I hope this will cause them to be more realistic,’' he continued. “People’s expectations [for alternative assessments] are too unrealistic, across the board. This is one signal for them to calm down.’'

Commissioner of Education Richard P. Mills said that, in light of the RAND report, Vermont would report only statewide results of the portfolio portion of the assessment. Officials orginally had planned to make public the results for each supervisory union, or group of districts.

Mr. Mills added that the state is committed to continuing with the new system and plans to seek a doubling of the program’s budget next year. Despite the reliability problem, he suggested, the assessment program has reaped other benefits, including improvements in curriculum and instruction.

“Teachers report that Vermont is changing curriculum, assessment, and instruction all at the same time, and for the better,’' Mr. Mills said. “They also have told me in great detail how terribly difficult this is, and how important it is for us to persist and get it right. We will.’'

Three-Part System

Created in 1988, the assessment program is Vermont’s first statewide testing program. It has drawn national attention at a time when a number of states and school districts are considering alternatives to traditional multiple-choice tests.

Under the program, 4th and 8th graders are assessed in writing and mathematics in three ways: a uniform test, which is a standardized test consisting of both multiple-choice and open-ended questions; a portfolio of classroom work completed throughout the year; and a “best piece’’ chosen by the student from the portfolio.

The program began on a pilot basis in the 1990-91 school year in 144 schools, and results were released statewide. It expanded to most, but not all, schools in the state last year.

Low Reliability Found

In a preliminary report on the first year of implementation, RAND researchers found that teachers and administrators considered the assessment time-consuming. But the study, conducted for the federally funded National Center for Research on Evaluation, Standards, and Student Testing, also found that educators felt the program had led to positive changes in curriculum and a greater understanding of student abilities. (See Education Week, Sept. 9, 1992.)

The new report, which examined how well the system fared as a measure of student performance, offers a far less positive picture.

Unlike traditional tests, in which computers register whether a student has filled in the correct bubble, the portfolios were scored by teachers, who evaluated each portfolio according to several criteria on a four-point scale.

For writing, the criteria consisted of purpose, organization, details, voice, and usage. For math, they were language of math, math representations, presentation, understanding of math, procedures, decisions, and outcomes.

The study used a standard statistical measure known as a reliability coefficient, which measures the extent to which two raters rank a student’s work the same. Under such a measure, no agreement would be zero, while total agreement would be 1.00.

The study found that reliability in the writing portfolios was quite low. Depending on the criterion on which the raters were evaluating, reliability coefficients ranged from 0.28 to 0.57. The reliability coefficients of the state’s uniform test in writing, by contrast, ranged from 0.67 in grade 8 to 0.75 in grade 4.

“For a variety of reasons, such as the variability of tasks used, it may be unrealistic to expect a portfolio program to reach as high a level of reliability as a standardized performance-assessment program’’ ... the report states. “However, the reliabilities obtained in Vermont in 1992 are sufficiently low to limit severely the uses to which the results can be put.’'

On the positive side, the study also found no evidence that teachers assigned higher or lower scores to their own students than did other raters.

In math, the study found similarly low reliability coefficients. Using a different measure of reliability, however, the study found that math raters assigned the same score to a portfolio about 60 percent of the time, compared with less than half the time for writing portfolios.

This discrepancy occurred, the report notes, because math scores tended to be concentrated at one or two points on the four-point scale.

Inadequate Training Seen

In examining possible reasons for the low levels of reliability, the RAND report suggests that the complex scoring scales may have contributed to the problem.

State officials said they are considering simplifying the scoring system.

“It’s difficult for evaluators to use all the criteria, and rate them all separately,’' said Douglas I. Tudhope, the chairman of the state board of education. “We have to look at that.’'

On a related point, the report also suggests that the training of the raters may have been inadequate.

Because the goal of the assessment program was to improve professional development as well as to evaluate student performance, Mr. Koretz pointed out, the state trained as many teachers as possible.

Other school systems that have conducted portfolio assessments, such as Pittsburgh, attained much higher levels of reliability by employing a relatively small number of highly trained raters, he noted.

But the effect of Vermont’s policy, Mr. Koretz said, was to produce a number of raters who may have been unsure of what high levels of performance looked like. This was particularly true in math, he said, where teachers accustomed to traditional forms of instruction were unfamiliar with such terms as “math representation,’' which indicates the extent to which students depict math concepts through graphs or drawings.

‘A Little Setback’

Vermont teachers involved in the project agreed that the training was inadequate.

“There wasn’t enough, and it came too late,’' said Clare Forseth, a math teacher at Marion Cross Elementary School in Norwich. “What has to happen is that teachers have to actually understand the criteria enough to have students understand it.’'

Commissioner Mills said that state officials would consider all of the report’s recommendations and make changes in the program. He said the state education department has asked RAND to evaluate the program again next year to determine if the reliability has improved.

But he stressed that the state is committed to portfolio assessment. He noted that the state board voted last week to request a doubling, to $841,000, of the program’s budget.

“This was a little setback,’' said Mr. Tudhope. “You can’t expect these things to be perfect the first time around.’'

Related Tags:

A version of this article appeared in the December 16, 1992 edition of Education Week as RAND Study Finds Serious Problems in Vt. Portfolio Program


Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
What’s Next for Teaching and Learning? Key Trends for the New School Year
The past 18 months changed the face of education forever, leaving teachers, students, and families to adapt to unprecedented challenges in teaching and learning. As we enter the third school year affected by the pandemic—and
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Webinar
How Data and Digital Curriculum Can Drive Personalized Instruction
As we return from an abnormal year, it’s an educator’s top priority to make sure the lessons learned under adversity positively impact students during the new school year. Digital curriculum has emerged from the pandemic
Content provided by Kiddom
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Judge's Temporary Order Allows Iowa Schools to Mandate Masks
A federal judge ordered the state to immediately halt enforcement of a law that prevents school boards from ordering masks to be worn.
4 min read
Iowa Gov. Kim Reynolds speaks to reporters following a news conference, Thursday, Aug. 19, 2021, in West Des Moines, Iowa. Reynolds lashed out at President Joe Biden Thursday after he ordered his education secretary to explore possible legal action against states that have blocked school mask mandates and other public health measures meant to protect students against COVID-19. Reynolds, a Republican, has signed a bill into law that prohibits school officials from requiring masks, raising concerns as delta variant virus cases climb across the state and schools resume classes soon. (AP Photo/Charlie Neibergall)
Education Hurricane Ida Deals New Blow to Louisiana Schools Struggling to Reopen
The opening of the school year offered teachers a chance to fully assess the pandemic's effects, only to have students forced out again.
8 min read
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021. Louisiana students, who were back in class after a year and a half of COVID-19 disruptions kept many of them at home, are now missing school again after Hurricane Ida. A quarter-million public school students statewide have no school to report to, though top educators are promising a return is, at most, weeks away, not months.
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021.
John Locher/AP
Education Massachusetts National Guard to Help With Busing Students to School
250 guard personnel will be available to serve as drivers of school transport vans, as districts nationwide struggle to hire enough drivers.
1 min read
Massachusetts National Guard soldiers help with logistics in this Friday, April 17, 2020 file photo, at a food distribution site outside City Hall, in Chelsea, Mass. Mass. Gov. Charlie Baker on Monday, Sept. 13, 2021, activated the state's National Guard to help with busing students to school as districts across the country struggle to hire enough drivers.
Massachusetts National Guard soldiers help with logistics in this Friday, April 17, 2020 file photo, at a food distribution site outside City Hall, in Chelsea, Mass.
Michael Dwyer/AP
Education FDA: ‘Very, Very Hopeful’ COVID Shots Will Be Ready for Younger Kids This Year
Dr. Peter Marks said he is hopeful that COVID-19 vaccinations for 5- to 11-year-olds will be underway by year’s end. Maybe sooner.
4 min read
Dr. Peter Marks, director of the Center for Biologics Evaluation and Research in the Food and Drug Administration, testifies during a Senate health, education, labor, and pensions hearing to examine an update from federal officials on efforts to combat COVID-19 on Capitol Hill in Washington on May 11, 2021. On Friday, Sept. 10, 2021, Marks urged parents to be patient, saying the agency will rapidly evaluate vaccines for 5- to 11-year-olds as soon as it gets the needed data.
Dr. Peter Marks, director of the Center for Biologics Evaluation and Research in the Food and Drug Administration, testifies during a Senate health, education, labor, and pensions hearing to examine an update from federal officials on efforts to combat COVID-19 on Capitol Hill in Washington on May 11, 2021.
Jim Lo Scalzo/AP