Education Opinion

Quality Reviews and the Fetishization of Data: A Fantasy

By Eduwonkette — February 07, 2008 4 min read
  • Save to favorites
  • Print

skoolboy returns for part II on Quality Reviews. You can find his first post here.

The year is 1975. Coach John Wooden of UCLA has just won his 10th NCAA men’s basketball championship in 12 years, a record that will likely never be matched in collegiate sports. Cambridge Associates sends Clive Wingtip to conduct a Quality Review of the UCLA program. Over the course of a day and a half, Wingtip talks with Coach Wooden, his assistant coaches, the players, and other staff, and observes the team practices. He also observes a collaborative activity: a meeting between Coach Wooden and his assistant coaches. The program is evaluated on five quality statements, each scored as either underdeveloped; underdeveloped with proficient features; proficient; well developed; or outstanding. Here’s a summary of the report he filed:

Quality statement 1: “The coach and staff consistently gather and generate data and use it to understand what each player knows and is able to do, and to monitor the player’s progress over time.”

There is evidence that Coach Wooden studies each individual, and his strengths and weaknesses, very carefully. But he does not rely on statistics collected during practice sessions or games to inform his judgment, and his observations are subjective, not objective. He does not measure performance and progress based on comparisons to similar schools. Overall Score: Underdeveloped

Quality statement 2: “The coach and staff consistently use data to understand each player’s next learning steps and to set suitably high goals for accelerating each student’s learning.”

The Coach and staff convey consistently high expectations to the players, and set specific goals for the team and for individuals. But Coach Wooden regularly played only 7 of the team’s 12 in games, and these same 7 practiced as a unit, suggesting that the reserves were not as important as the regulars. Coach Wooden occasionally displays the soft bigotry of low expectations. He states, “There is nothing wrong with the other fellow being better than you are, as long as you did everything you possible could to prepare yourself for the competition. That is all you have control over. It may be that the other fellow’s level of competency is simply higher than yours. That doesn’t make you a loser.” Overall Score: Underdeveloped with Proficient Features

Quality statement 3: “The program aligns its work, strategic decisions and resources, and effectively engages players, around its goals and plans for accelerating players’ learning.”

Although there was evidence of individualized instruction, the Coach and staff did not use objective team and individual data to plan for and provide this instruction. Team members trusted and respected Coach Wooden. Overall Score: Proficient

Quality statement 4: “The program has structures for monitoring and evaluating each player’s progress throughout the year and for flexibly adapting plans and practices to meet its goals for accelerating learning.”

Coach Wooden relies heavily on repetition during the season. One player said, “He never talks about strategy, statistics, or plays but rather about people and character.” The program did not rely on objective measures to assess progress towards goals, such as the final score of games or written tests, and did not have a playbook. Coach Wooden frequently adjusted the plans for each practice, and made notes after each practice about adjustments; but these notes were based on subjective judgments, not hard data on performance. Overall Score: Underdeveloped with Proficient Features

What’s wrong with this picture? How does the man voted Coach of the Century by ESPN receive a quality review rating of “underdeveloped”? It’s all about the quality criteria, which privilege using data to make decisions about how to help student/athletes to learn and develop over simply making good decisions about teaching and learning. I refer to this as the “fetishization of data.” In the Quality Review game, quantitative performance data have become an end in themselves, rather than a means to an end. One of the most cutting insults that one social scientist can hurl at another is to label another’s research as a “data dump.” In some school districts which have embraced external quality reviews, compiling notebooks full of undigested data has become a substitute for thoughtful analysis of a reasonably small number of important themes and problems.

What’s the solution? I’d start by broadening the definition of data beyond quantitative performance measures. There’s no doubt in my mind that Coach John Wooden relied heavily on data to inform the design of his practices, and his approach to cultivating the talents of his players and his team. But those data took the form of the systematic observations and judgments of an expert practitioner. I’d also seek to evaluate schools on the basis of the quality of the teaching within them, not whether the educators in a school arrived at their teaching practices via the analysis of quantitative performance data.

“Don’t mistake activity for achievement…If you spend too much time learning the tricks of the trade, you may not learn the trade.” (John Wooden)

The opinions expressed in eduwonkette are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Student Well-Being Webinar
Attendance Awareness Month: The Research Behind Effective Interventions
More than a year has passed since American schools were abruptly closed to halt the spread of COVID-19. Many children have been out of regular school for most, or even all, of that time. Some
Content provided by AllHere
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
School & District Management Webinar
Ensuring Continuity of Learning: How to Prepare for the Next Disruption
Across the country, K-12 schools and districts are, again, considering how to ensure effective continuity of learning in the face of emerging COVID variants, politicized debates, and more. Learn from Alexandria City Public Schools superintendent
Content provided by Class

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Schools Get the Brunt of Latest COVID Wave in South Carolina
In the past few weeks, South Carolina has set records for COVID-19 hospitalizations and new cases have approached peak levels of last winter.
4 min read
Two Camden Elementary School students in masks listen as South Carolina Gov. Henry McMaster talks about steps the school is taking to fight COVID-19, Wednesday, Sept. 15, 2021, in Camden, S.C. McMaster has adamantly and repeatedly come out against requiring masks in schools even as the average number of daily COVID-19 cases in the state has risen since early June. (AP Photo/Jeffrey Collins)
Education More States Are Requiring Schools to Teach Native American History and Culture
Advocates say their efforts have gained some momentum with the nation’s reckoning over racial injustice since the killing of George Floyd.
3 min read
A dancer participates in an intertribal dance at Schemitzun on the Mashantucket Pequot Reservation in Mashantucket, Conn., Saturday, Aug. 28, 2021. Connecticut and a handful of other states have recently decided to mandate students be taught about Native American culture and history. (AP Photo/Jessica Hill)
Education Judge's Temporary Order Allows Iowa Schools to Mandate Masks
A federal judge ordered the state to immediately halt enforcement of a law that prevents school boards from ordering masks to be worn.
4 min read
Iowa Gov. Kim Reynolds speaks to reporters following a news conference, Thursday, Aug. 19, 2021, in West Des Moines, Iowa. Reynolds lashed out at President Joe Biden Thursday after he ordered his education secretary to explore possible legal action against states that have blocked school mask mandates and other public health measures meant to protect students against COVID-19. Reynolds, a Republican, has signed a bill into law that prohibits school officials from requiring masks, raising concerns as delta variant virus cases climb across the state and schools resume classes soon. (AP Photo/Charlie Neibergall)
Education Hurricane Ida Deals New Blow to Louisiana Schools Struggling to Reopen
The opening of the school year offered teachers a chance to fully assess the pandemic's effects, only to have students forced out again.
8 min read
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021. Louisiana students, who were back in class after a year and a half of COVID-19 disruptions kept many of them at home, are now missing school again after Hurricane Ida. A quarter-million public school students statewide have no school to report to, though top educators are promising a return is, at most, weeks away, not months.
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021.
John Locher/AP