Education Funding

Teacher-Evaluation Logistics Challenge States

By Stephen Sawchuk — April 26, 2011 8 min read

Rounding the corner on the design of new teacher-evaluation plans, states and districts are beginning to wrestle with the significant technical and logistical hurdles for transforming their blueprints into reality.

In the coming months, more states—especially those that won grants through the $4 billion federal Race to the Top initiative—are expected to put out requests for proposals for such details as overhauls to the data systems that store student and teacher information; the provision of “value added” analyses of teacher performance; and the reporting and professional development that help teachers and principals use the information from the systems.

In sum, states and districts must construct integrated systems for teacher-performance management—no small challenge in a field in which few good examples exist.

“States and districts are going to need considerable amounts of technical expertise,” said Brian M. Stecher, the associate director of RAND Education at the Santa Monica-based RAND Corp., a nonprofit policy-research organization.

The situation is complicated by the fact that the marketplace of contractors, vendors, and providers that states must rely on to fulfill their plans is still evolving.

To give just one example, many of the technical experts on such elements as the statistical modeling of value-added teacher estimates don’t work for traditional education publishers or test companies, but for organizations now primarily known for research or program evaluation.

“It’s a very active conversation, I can tell you,” said Brian P. Gill, a senior fellow at Mathematica Policy Research, a Princeton, N.J.-based organization.

Known for conducting large-scale, federally sponsored evaluations of education programs, Mathematica recently expanded into a technical-assistance role helping such partners as the District of Columbia and Pittsburgh school districts implement teacher-evaluation systems.

“We’re observing changes happening in the field, and we’re trying to figure out what our appropriate place is, where we can provide the most value,” Mr. Gill said.

As Race to the Top grants begin to mature, deadlines states have set in the teacher and leader sections of their plans are starting to bear down on them.

Ohio’s new teacher-evaluation system goes into effect next school year for its 26 participating districts. The state is still working out details of the student-growth component, officials there say.

Policy Momentum

Under Georgia’s plan, the state will likely put out a request for proposals this summer to secure a contractor to supply value-added estimates based on its state-test data to include in new teacher-evaluation systems.

And in February, the Florida education department awarded the publishing giant Houghton Mifflin Harcourt a contract to craft model teacher- and leader-observation protocols based on the work of Douglas Reeves and Robert Marzano, two well known professional-development consultants, and to help school districts implement those models or locally designed alternatives.

Houghton Mifflin Harcourt declined to disclose the contract’s worth, and the Florida education department did not respond to inquiries. But in any case, considerable money is at stake. Each of the 12 Race to the Top winners, save Massachusetts, is devoting more than half the state share of its grant to contractors. Of the four key areas of focus in the program, the teacher- and leadership-effectiveness piece is likely to make up the biggest portion of spending because of the large capacity-building needs in those areas.

The policy momentum for those activities extends far beyond the federal grant program, which was financed by the American Recovery and Reinvestment Act. Legislation overhauling state evaluation guidelines are pending in several statehouses; the renewal of the Elementary and Secondary Education Act looms; and private philanthropies such as the Seattle-based Bill & Melinda Gates Foundation are also pouring in thousands of dollars to support changes to states’ and districts’ teacher-talent structures.

“The opportunity in Florida was unique because of the funding, and the mandate, and our capability to deliver something,” Dave R. West, the senior vice president of Houghton Mifflin Harcourt Education Consulting Services, said of the company’s winning bid in that state. “Florida is a bellwether state, but all states are going to be interested in this kind of solution, in teacher and leader effectiveness.”

Building Systems

With money and policy prescriptions in hand, it’s easy to find reasons why states and districts are seeking outside expertise on these most complicated of issues.

Take value-added data, for instance. Even for those states that have experience measuring student growth as part of their school accountability systems, making the leap to individual teacher estimates is not as simple as flipping a switch. In general, experts say that drilling down to classroom and teacher value-added estimates demands data of far higher quality than is typical.

“It is hugely, hugely challenging to get all of the data accurate, before you even start talking about the analytical process,” said William L. Sanders of the Cary, N.C.-based SAS Institute and one of the pioneers of value-added modeling. “Particularly when you’re doing this at the classroom level, you’ve got to make sure the right students get attributed to the right teachers.”

One problem is more philosophical. All value-added models have degrees of error associated with them, and while researchers say that additional years of data help make the estimates more precise, debate continues within the research community about other aspects, such as whether the models should control for student demographic characteristics.

In short, there is no “best” value-added model, and two distinct specifications will yield somewhat different estimates.

“You can begin with the same data, but if you make different choices about how much to weigh each element, you will come out with different estimates of the outcomes of interest,” noted Mr. Stecher of RAND.

Observers say it will be equally hard to put in the other pieces of new teacher-evaluation systems, such as multiple observations of teachers’ classroom skills. There again, states and districts will wrestle with how the observations are conducted and how the resulting information will be collected, audited, and stored.

Unlike value-added statistics, such information gives teachers guidance for improvement—but only if used with care.

“The real opportunity will come from integrating information from value-added with much richer information for what teachers are doing in the classroom,” said Mr. Gill of Mathematica. “And that part, frankly, is a lot harder to collect, it’s more expensive to collect, and it’s harder to make systematic and fair.”

It also means putting together the pieces, not necessarily an easy task because contractors and providers generally have expertise only in certain areas. One provider may specialize in data “warehousing,” another in the development of a value-added model, another in professional development for an evaluation framework, and yet another in the packaging, reporting, and dissemination of information to teachers and administrators in a simple, easy-to-understand format.

“All of these different parts in the districts’ data systems need to be able to talk to each other. If your data isn’t well organized, it delays things and makes it slower to assemble the information,” said Robert H. Meyer, a research professor and director of the Value-Added Research Center at the University of Wisconsin-Madison. “For teachers and principals, this information is most useful when they get it asap.”

New Roles?

The need for design help is also blurring the lines that have traditionally separated organizations engaged in research or evaluative roles from technical-assistance providers.

Mathematica has moved in the technical-assistance direction in part because so many districts lack the infrastructure to begin moving on systems for measuring teacher effectiveness, Mr. Gill said.

“We’ve gotten into this in part because on the big [U.S. Department of Education] projects, there was a need—school districts that wanted to make decisions on the information and didn’t have the internal capacity to do it,” he said.

At the same time, Mr. Gill underscored that serving as a program evaluator still is the organization’s bread-and-butter mission. “We’re a research and evaluation firm, and that’s the great majority of what we do,” Mr. Gill said.

The UW-Madison center has expanded its technical assistance since the 1990s. It recently won a contract to help Los Angeles carry out a pilot program using value-added data, a venture worth up to $1.5 million, according to The Los Angeles Times. Mr. Meyer said that his group balances long-term research goals on value-added issues with shorter-term plans to help states and districts update their systems over time, and finally, with responding to districts’ real-time demands for teacher-performance information.

“In some sense, we split our year into a research phase and a production phase,” he said. “When we do research, we do research; when we’re in production, we’re trying to have all the features of a business in terms of timeliness and product quality.”

Mr. Stecher, meanwhile, thinks that nonprofits such as RAND could have yet another role to play: that of a third-party auditor that helps, for instance, analyze how a teacher-evaluation system affects teachers’ behavior.

“Are teachers still making decisions in the best interests of good instruction, or are they trying to game the system to do what they think will artificially raise their value-added scores?” he said. “A lot of these things can’t be known until after the fact.”

Large education publishers are also staking claims in the emerging teacher-effectiveness arena. In January, the division of Houghton Mifflin Harcourt that Mr. West helps direct acquired the Leadership and Learning Center, a consultancy founded by Mr. Reeves.

With its experience with value-added information, the privately held SAS Institute might benefit from the expansion of such models. But the rapid pace of expansion is nevertheless cause for some concern, Mr. Sanders said.

“People who have not attempted to do this before don’t realize how complicated and complex this is going to be,” he said. “It’s especially true for researchers who are used to getting one data set before they do their analysis for publication. You get into a production environment where there are deadlines, and it becomes a whole different set of issues.”

Coverage of the American Recovery and Reinvestment Act is supported in part by grants from the William and Flora Hewlett Foundation, at www.hewlett.org, and the Charles Stewart Mott Foundation, at www.mott.org.
A version of this article appeared in the April 27, 2011 edition of Education Week as Building Systems for Evaluation of Teachers Poses Challenge

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
Interactive Learning Best Practices: Creative Ways Interactive Displays Engage Students
Students and teachers alike struggle in our newly hybrid world where learning takes place partly on-site and partly online. Focus, engagement, and motivation have become big concerns in this transition. In this webinar, we will
Content provided by Samsung
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Classroom Technology Webinar
Educator-Driven EdTech Design: Help Shape the Future of Classroom Technology
Join us for a collaborative workshop where you will get a live demo of GoGuardian Teacher, including seamless new integrations with Google Classroom, and participate in an interactive design exercise building a feature based on
Content provided by GoGuardian
School & District Management Live Online Discussion A Seat at the Table With Education Week: What Did We Learn About Schooling Models This Year?
After a year of living with the pandemic, what schooling models might we turn to as we look ahead to improve the student learning experience? Could year-round schooling be one of them? What about online

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Funding Biden Pitches 41 Percent Spending Increase for Education Next Year on Top of COVID-19 Aid
The president wants nearly $103 billion for the Department of Education, although history indicates Congress won't approve that request.
4 min read
Conceptual image of money, a mask, and the American flag.
Collage by Laura Baker/Education Week (Images: HAKINMHAN/iStock/Getty and Cimmerian/E+)
Education Funding Biden Infrastructure Plan Calls for $100 Billion for School Construction, Upgrades
President Joe Biden's $2 trillion American Jobs Plan would also fund replacement of lead pipes and expand broadband internet access.
4 min read
President-elect Joe Biden speaks at The Queen Theater on Dec. 29, 2020, in Wilmington, Del.
President-elect Joe Biden speaks at The Queen Theater on Dec. 29, 2020, in Wilmington, Del.
Andrew Harnik/AP
Education Funding Miguel Cardona Releases $912 Million for Puerto Rico's Schools, Easing Trump Restrictions
Puerto Rico has regained access to hundreds of millions of dollars for education to address the fallout of COVID-19 and other needs.
2 min read
Students arrive at the Ramon Marin Sola primary school for the first time in nearly a year amid the COVID-19 pandemic as some public schools reopen in San Juan, Puerto Rico on March 10, 2021.
Students arrive at the Ramon Marin Sola primary school for the first time in nearly a year amid the COVID-19 pandemic as some public schools reopen in San Juan, Puerto Rico on March 10.
Danica Coto/AP
Education Funding School Budgets: Why They're Not As Bad As Predicted
Revenue projections are up, but districts aren't out of the woods. Seven questions answered about the evolving landscape for budgets.
11 min read
Image shows a businessman searching for new revenue in unchartered waters standing on a compass among several waves.
iStock/Getty Images Plus