College & Workforce Readiness

Experts See Challenges Ahead in Creating College-Rating System

By Caralee J. Adams — September 30, 2013 6 min read
  • Save to favorites
  • Print


President Obama has called for a new college-rating system to provide the public with information about value and affordability in higher education, but it’s not clear yet how those concepts will be measured. Yet, the administration wants to have a draft of the system completed by the fall of 2014.

That will make for a challenging and controversial year ahead, according to higher education experts gathered for a panel here this past weekend. The panel was part of a conference sponsored by the Education Writers Association.

“This is something we will be wrestling with for the rest of this administration,” said Terry Hartle, senior vice president of the American Council of Education. “They have the authority to do this. You should bet your bottom dollar that they will do this. You don’t have the President of the United States give three speeches and suddenly decide it’s too hard to do. They will do it. The extent to which it will be done well is very much an open question.”

Initially, the administration suggested the rating system would be based on access, such as percentage of students receiving Pell grants; affordability, such as average tuition, scholarships, and loan debt; and outcomes, such as graduation and transfer rates, graduate earnings, and advanced degrees of college graduates.

How those criteria will be defined is open for discussion. The administration unveiled the idea in broad terms and said it would welcome feedback from the higher education community.

Zakiya Smith, a strategy director at the Lumina Foundation, said it is good that the metrics were not hammered out by policymakers and just announced, but that experts from colleges and universities are being asked for their input.

“I’m cautiously optimistic,” she said. “I think it’s important to give students some context about choosing college and give an authoritative source, which the government has the potential to do.”

(The Lumina Foundation also supports Education Week coverage of P-16 alignment.)

Outcomes Data is Limited

While some welcome giving the public more information about colleges, the idea of linking performance to the amount of federal student aid that higher education institutions receive will likely spark some debate, the panelists noted. The U.S. Department of Education plans to move ahead in creating the rating system, but tying it to federal aid would require congressional action.

Smith said the rating system could be positive for transparency, but the same system may not work for accountability.

Hartle said the biggest concern among college and university administrators is that poor quality data will be used in the rating system.

“We don’t know where this data will come from,” he said. If the ratings are linked to serious consequences, such as federal funding, Hartle said it was vital that the data be very accurate. The outcome data that the federal government currently collects is “quite limited.”

Once measures are agreed upon, the Education Department will need to decide how to weigh various factors. Hartle suggested proposed formulas be shared with higher education practitioners for peer review and advice before finalizing a system.

“Developing a rating system is hard,” he said. “These things don’t turn out real well right off the bat.”

For instance, the methodology used for the Best Colleges rankings produced by U.S. News and World Report has evolved over the past 35 years, Hartle said. While the administration is proposing a “rating” system, the reality is that outside sources eventually will likely take the data and turn it into a “ranking” that gives a numerical order, Hartle suggests.

To be fair, the new system should compare similar types of institutions, such as community colleges or four-year state universities, to each other because they have different missions and outcomes, the panelists said. Consideration should be given to even comparing outcomes of programs within institutions because there can be such variation between engineering and music, for instance, suggested Matt Reed, the vice president for academic affairs with Holyoke Community College in Holyoke, Mass. Otherwise, he said, schools may drop programs where students can’t get lucrative jobs.

Regional Variations

Reed also noted that community colleges differ widely across the country. They are often affected by the local economies and the concentration of colleges in a region. For instance, in North Dakota there are fewer four-year colleges so better prepared students may attend community colleges, improving graduation rates compared with a college in a more competitive area. Also, the regional job outlook and cost of living could influence student outcomes as measured by employment and salary, Reed said.

“When comparing Alaska to North Dakota to Massachusetts to California, the contexts are so utterly different that to put everyone on the same grid, I think, will lead to really significant distortions,” he said.

Current graduation data often misses nontraditional students, such as those who transfer or take longer than usual to get a degree. The current measure was created in the 1990s based on young college students just out of high school, going to school full-time and living on campus and those traditional students now make up a minority of the college-going population.

The panelists said that ideally, a “unit record system: would be devised that tracks every student from institution to institution to better capture student outcomes. However, it may take up to 10 years to have reasonable data, and the notion of more “big government databases” is not very popular now, said Hartle. “This is the solution, but it is not an easy one,” he said, noting that it likely trigger concerns over privacy and who would maintain such a vast database.

Smith mentioned using graduation information on Pell grant recipients as a possible measure of outcomes. All the panelists discussed the need for any new system to capture the performance of colleges in serving disadvantaged students, so there would be an incentive for colleges to serve students from all backgrounds.

“I’m afraid the data we will use for the rating system will be from data that are currently available - for better or worse,” said Hartle.

Start With Four-Year Institutions?

Smith said college administrators have ways of portraying their success to their governing boards and they should be willing to share those strategies with the Education Department as the system is being developed.

Hartle suggested that the Obama administration first focus on four-year institutions where the issue of affordability is more acute than at community colleges. He also said it was important to develop a rating system with criteria that would not change from one administration to the next so it would be possible to follow the performance of colleges over time.

U.S. Sen. Elizabeth Warren, a Massachusetts Democrat who spoke separately at the EWA conference, said she isn’t clear about the intent of the rating system but she would support the administration’s plan if the core is rating colleges on cost. In her remarks, Warren suggested several ways to reform the student loan and student financial aid programs to reduce costs, which said is the main issue to confont.

“I applaud the president for standing up and talking about higher education,” she said on Saturday. “It’s time for the federal government to use its muscle to make sure families can get an affordable college education.”

Related Tags:

A version of this news article first appeared in the College Bound blog.