Opinion
Education Teacher Leaders Network

Does Grading Bias Apply to Education Reports?

By Bob Williams — December 16, 2009 6 min read
  • Save to favorites
  • Print

In my efforts to increase student interest in understanding probability, I often do a magic trick where I repeatedly shuffle a deck of cards and have three different students pick a card. Using the counting principle, students see that if we went through this process more than 132,000 times, theoretically I would guess the selected card of all three students only once. Of course, since I know how to trick my students into selecting the three cards I want them to select, I get the answers I want every time.

Last month, the U.S. Chamber of Commerce, the American Enterprise Institute, and the Center for American Progress published a damning report on education titled Leaders and Laggards that I believe uses similar sleight of hand. The report is presented as a fair, research-based, and data-supported assessment to determine how states are doing in innovation of education. What I see under the surface, however, is a strong bias for Teach for America and alternative-certification programs and against experienced teachers, teacher tenure, and teacher unions.

The report’s attention-getting strategy is to assign grades, with states receiving individual letter grades for seven broad educational categories. Each category comprises several indicators that are given numerical values and tabulated to produce a numerical score for the category, which is then converted to a letter grade.

But the data used in Leaders and Laggards’ seems highly selective. For example, one indicator under the “Staffing: Hiring and Evaluation” category is the state’s use of “national programs to recruit nontraditional teachers,” namely Teach for America, The New Teacher Project, and Troops to Teachers. A state’s score for this indicator is influenced heavily by whether or not it uses teachers from the TNTP and TFA. If it does, it gets 100 points (an A); if it doesn’t, it gets 59. (That score is then combined with a rating given for its relative number of Troops to Teachers.)

But the quality or effectiveness of teachers hired through these programs does not influence the grade. The authors simply note that these organizations “have demonstrated a strong record, based on independent research, of bringing nontraditional applicants into the classroom. …” But, from a different perspective, I think you could also argue that states reliant on these programs have not adequately managed their pipeline of qualified teachers and are overly willing to place people with limited formal teacher-training and short-term commitments into high-needs classrooms.

Few would argue against the proposition that every state should be committed to placing the most competent and effective teachers into the classroom. If TFA and TNTP teachers are the strongest candidates available, they should get the job and hopefully develop a passion to improve their practice and increase their commitment to many years of teaching. But grading states numerically based on a blanket commitment to TFA and TNTP, instead of the effectiveness and quality of teachers hired, seems questionable. And by the way, where’s the indicator for innovative practices in supporting and developing all teachers over the course of their careers?

Likewise, the “Hiring and Evaluation” category includes an indicator for alternative certification. The report rewards states with higher levels of alternatively certified teachers and penalizes states with low levels. A separate indicator rates the states on the “strength” of their alternative-certification systems (in terms of admissions standards and accountability), but no attempt is made to determine the actual quality or the effectiveness of the alternatively certified teachers. Are they just assumed to be better for some reason?

As with several other indicators, the ten states with the highest percentage of alternative-route teachers receive an A, 100 points; the next ten states receive a B, and so forth. The process thus guarantees an alarming conclusion, since the grading is designed to give 40 percent of the states Ds or Fs. I have won multiple awards for effective teaching, but if I assessed my students using this assessment process, I would be on a plan of improvement next week.

And there’s another problem with this indicator. The report assigns a degree of accuracy to the alternative- certification data that I don’t think can be said to exist. The percentage of alternative-certificate teachers in a state was based on an estimate from the 2007-08 Schools and Staffing Survey from the U.S. Department of Education. This estimate comes from a sampling of teachers and principals from each state who completed a mail-in survey questionnaire. As I understand it, SASS surveyed on average only three to eight teachers at schools selected for its sample. So Leaders and Laggards is assigning letter grades on the basis of self-reported survey data. What was the margin of error for the state survey results?

To take another example, the report also grades states in the broad category of “Staffing: Removal of Ineffective Teachers.” This assessment is based entirely on the SASS survey of school principals. States are graded on the basis of the responses that surveyed principals gave to a set of questions on “barriers” to removing ineffective teachers. If a high percentage of principals in a state reported such barriers—e.g., “teacher associations and unions,” “teacher tenure,” “personnel policies” “dismissal is too stressful”—that state received a low grade for the category.

This is another case where the report’s methodology seems calculated to generate sensationalistic results. Again the data is sorted to give the top ten states an A, the next ten a B, the next ten a C, and the remaining states Ds and Fs. But the actual data from the principal survey is surprisingly tightly grouped. A state score of 65 (average) received an A in the report. State scores of 63 and 64 received Bs and a state score of 62 received a C. That is a very steep grading curve, especially for data based on a survey.

More troubling is the fact that the report makes no attempt to determine how many principals actually took steps to remove ineffective teachers or whether the principals in the survey were simply imagining or presuming what barriers they might someday experience. Not to mention that many of the so-called “barriers” referenced actually have legitimate—some might say even say valuable—purposes within a school system. Why weren’t those measured?

The authors of this report appear to have stacked the deck in order to achieve their underlying goal. They believe that improving education requires weakening teacher tenure and union influence while supporting alternative certification and national programs to place inexperienced people (through TFA and TNTP) into teaching positions with minimal training.

The authors’ biased use of data does not inspire me to look to their organizations for solutions to our educational challenges. Outside groups that lack teacher perspectives, distort data and have a pre-determined agenda may have something meaningful to offer. But I’d estimate that chance to be about the same as correctly guessing in order, three randomly picked cards from the deck (1/132600), without resorting to a magician’s trick.

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Classroom Technology Webinar
Academic Integrity in the Age of Artificial Intelligence
As AI writing tools rapidly evolve, learn how to set standards and expectations for your students on their use.
Content provided by Turnitin
Recruitment & Retention Live Online Discussion A Seat at the Table: Chronic Teacher Shortage: Where Do We Go From Here?  
Join Peter DeWitt, Michael Fullan, and guests for expert insights into finding solutions for the teacher shortage.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Reading & Literacy Webinar
The Science of Reading: Tools to Build Reading Proficiency
The Science of Reading has taken education by storm. Learn how Dr. Miranda Blount transformed literacy instruction in her state.
Content provided by hand2mind

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Letter to the Editor EdWeek's Most-Read Letters of 2022
Here are this year’s top five Letters to the Editor.
1 min read
Education Week opinion letters submissions
Gwen Keraval for Education Week
Education In Their Own Words Withstanding Trauma, Leading With Honesty, and More: The Education Stories That Stuck With Us
Our journalists highlight why stories on the impact of trauma on schooling and the fallout of the political discourse on race matter to the field.
4 min read
Kladys Castellón prays during a vigil for the victims of a mass shooting at Robb Elementary School in Uvalde, Texas, on Tuesday, May 24, 2022.
Kladys Castellón prays during a vigil for the victims of a mass shooting at Robb Elementary School.
Billy Calzada/The San Antonio Express-News via AP
Education In Their Own Words Masking, Miscarriages, and Mental Health: The Education Stories That Stuck With Us
Our reporters share the stories they wrote that rose above the fray—and why.
5 min read
Crystal Curtis and her son, Jordan Curtis, outside their home in Plano, Texas. Crystal, a healthcare professional whose son attends school in Plano talks about the challenges of ensuring quality schooling, her discomfort with the state and district’s rollback of mandatory masking, and the complications of raising a Black child in a suburban district as policies shift.
Crystal Curtis and her son, Jordan Curtis, outside their home in Plano, Texas. Crystal, a healthcare professional whose son attends school in Plano talks about the challenges of ensuring quality schooling, her discomfort with the state and district’s rollback of mandatory masking, and the complications of raising a Black child in a suburban district as policies shift.
Allison V. Smith for Education Week
Education Opinion The Top 10 Rick Hess Straight Up Columns of 2022
NAEP, pre-K, who decides what gets taught. Those are among the most popular or impactful posts of the year.
2 min read
Image shows a multi-tailed arrow hitting the bullseye of a target.
DigitalVision Vectors/Getty