Teaching Profession

Class Composition Can Bias English Teachers’ Observation Scores, Study Finds

By Stephen Sawchuk — January 22, 2016 2 min read
  • Save to favorites
  • Print

Observations of teachers—usually the most prominent component of teacher-evaluation systems—can carry significant sources of bias, potentially penalizing English/language arts teachers of lower-achieving students, concludes a recent research study.

The paper, by Matthew Steinberg of the University of Pennsylvania and Rachel Garrett of the American Institutes of Research, was published in the January edition of the journal Educational Evaluation and Policy Analysis.

The analysis provides more evidence that, despite the widespread concern about test-score-based ratings of teachers, observations of teachers are just as susceptible to error. It deepens earlier findings by looking at the topic at a more granular level, and by showing that the findings are consistent across several analytical samples.

For the study, the researchers examined observation scores of some 834 teachers, who were rated by trained reviewers on Charlotte Danielson’s Framework for Teaching. (The data come from the Bill & Melinda Gates Foundation’s Measures of Effective Teaching study, which examined several different measures of teacher performance.)

The researchers found that ELA teachers of higher-achieving students were more likely to get higher scores on the “classroom environment” domain of the Framework for Teaching, which considers teachers’ classroom management and their ability to create a respectful learning environment, among other goals. Overall, an English teacher with a class whose incoming achievement was a standard deviation higher would get his or her score pushed up by about a third of a standard deviation.

The study found that the pattern was present even among a subset of teachers who were randomly assigned to a class of students—an important distinction because that research technique helps filter out other potential sources of bias.

Higher-performing students are less likely to have behavior infractions, the authors postulate. “It sort of makes sense,” Steinberg said. “If you have better behaved students, it’s probably easier to work with them and to develop that rapport.”

Now here’s the interesting thing. The bias did not show up on all of the FFT domains. Teachers’ ratings on things directly related to instructional technique—whether they ask students probing questions and use assessments for instruction—weren’t related to students’ prior achievement. And math teachers generally didn’t seem to get the same boost from having better-performing students. The authors postulate that math instruction, by its very nature, tends to be more direct and to use fewer groups, conversations, and other techniques that rise and fall on student interactions.

What does this mean for policy? Should states and districts chuck out pieces of these observation protocols that seem susceptible to this kind of bias? Steinberg doesn’t think so, since the information from these systems can still be used as a tool to help teachers improve overall.

But, if these indicators are going to factor into a rating that might affect teacher pay, promotion, or job security, then some tweaks might be needed—like using a rolling average over several years to help even out swings in performance due to classroom composition.

“Practitioners and policymakers really need to meet in the middle on this,” he said.

See more on teacher evaluation:

A version of this news article first appeared in the Teacher Beat blog.

Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Attendance Awareness Month: The Research Behind Effective Interventions
More than a year has passed since American schools were abruptly closed to halt the spread of COVID-19. Many children have been out of regular school for most, or even all, of that time. Some
Content provided by AllHere
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Ensuring Continuity of Learning: How to Prepare for the Next Disruption
Across the country, K-12 schools and districts are, again, considering how to ensure effective continuity of learning in the face of emerging COVID variants, politicized debates, and more. Learn from Alexandria City Public Schools superintendent
Content provided by Class

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Teaching Profession Opinion ‘A Culture of Care’: How Schools Can Alleviate Educator Stress This Year
It takes more than deep breathing to alleviate the stress teachers feel. Here's how to get to the root cause.
Sean Slade & Alyssa Gallagher
6 min read
shutterstock 740616958 resized
Teaching Profession Reported Essay Students Aren’t the Only Ones Grieving
Faced with so many losses stemming from the pandemic, what can be done to help teachers manage their own grief?
4 min read
Conceptual Illustration
Pep Montserrat for Education Week
Teaching Profession We Feel Your Grief: Remembering the 1,000 Plus Educators Who've Died of COVID-19
The heartbreaking tally of lives lost to the coronavirus continues to rise and take a steep toll on school communities.
3 min read
090321 1000 Educators Lost BS
Education Week
Teaching Profession Letter to the Editor Educators Have a Responsibility to Support the Common Good
A science teacher responds to another science teacher's hesitation to take the COVID-19 vaccine.
1 min read