Opinion Blog


Rick Hess Straight Up

Education policy maven Rick Hess of the American Enterprise Institute think tank offers straight talk on matters of policy, politics, research, and reform. Read more from this blog.

Policy & Politics Opinion

The 2021 RHSU Edu-Scholar Public Influence Scoring Rubric

By Rick Hess — January 05, 2021 10 min read
  • Save to favorites
  • Print

Tomorrow, I’ll be unveiling the 11th annual 2021 RHSU Edu-Scholar Public Influence Rankings, recognizing the 200 university-based scholars who had the biggest influence on educational practice and policy last year. Today, I want to run through the methodology used to generate those rankings.

Given that more than 20,000 university-based faculty in the U.S. are researching education, simply making it onto the Edu-Scholar list is an accomplishment. The list is comprised of university-based scholars who focus primarily on educational questions (with “university-based” meaning a formal university affiliation). Scholars who do not have a formal affiliation on a university website are ineligible.

The 150 finishers from last year automatically qualified for a spot in this year’s Top 200, so long as they accumulated at least 10 “active points” in last year’s scoring. (This is a gauge of current activity and so includes all categories except Google Scholar and Book Points, which are measures of career-long achievement.) The automatic qualifiers were then augmented by “at-large” additions chosen by the RHSU Selection Committee, a disciplinarily, methodologically, and ideologically diverse group of accomplished scholars. All Selection Committee members had automatically qualified for this year’s rankings.

I’m indebted to the RHSU Selection Committee for its assistance and want to acknowledge the 2020 members: Donna Ford (Ohio State), Dan Goldhaber (U. Washington), Sara Goldrick-Rab (Temple), Jay P. Greene (U. Arkansas), Margaret Grogan (Chapman), Eric Hanushek (Stanford), Shaun Harper (USC), Douglas N. Harris (Tulane), Carolyn J. Heinrich (Vanderbilt), Jeffrey R. Henig (Columbia), Robert Kelchen (Seton Hall), Helen F. Ladd (Duke), Gloria Ladson-Billings (U. Wisconsin), Marc Lamont Hill (Temple), Susanna Loeb (Brown), Bridget Terry Long (Harvard), Ernest Morrell (Notre Dame), Pedro Noguera (USC), Robert C. Pianta (U. Virginia), Jonathan Plucker (Johns Hopkins), Marcelo Suarez-Orozco (U. Mass Boston), Carol Tomlinson (U. Virginia), Jacob L. Vigdor (U. Washington), Kevin G. Welner (CU Boulder), Martin West (Harvard), Patrick J. Wolf (U. Arkansas), Yong Zhao (U. Kansas), and Jonathan Zimmerman (U. Penn).

OK, so that’s how the Top 200 list was compiled. How were the actual rankings calculated? Each scholar was scored in nine categories, yielding a maximum possible score of 200. Scores are calculated as follows:

Google Scholar Score: This figure gauges the number of widely cited articles, books, or papers a scholar has authored. For this purpose, I use each scholar’s “h-index.” This is a useful, popular way to measure the breadth and impact of a scholar’s work. It involves tallying a scholar’s works in descending order of how often each is cited and then identifying the point at which the number of oft-cited works exceeds the cite count for the least-frequently cited. For instance, a scholar who had 20 works that were each cited at least 20 times, but whose 21st most-frequently cited work was cited just 10 times, would score a 20. The measure recognizes that bodies of scholarship matter greatly for influencing how important questions are understood and discussed. The search was conducted using the advanced search “author” filter in Google Scholar. For those scholars who have created a Google Scholar account, their h-index was available at a glance. For those scholars without a Google Scholar account, a hand search was used to calculate their score while culling out works by other, similarly named, individuals. Points were capped at 50. (This search was conducted on Dec. 15-16.)

Book Points: A search on Amazon tallied the number of books a scholar has authored, co-authored, or edited. Scholars received 2 points for a single-authored book, 1 point for a co-authored book in which they were the lead author, a half-point for co-authored books in which they were not the lead author, and a half-point for any edited volume. The search was conducted using an “Advanced Books Search” for the scholar’s first and last name. (On a few occasions, a middle initial or name was used to avoid duplication with authors who had the same name, e.g., “David Cohen” became “David K. Cohen.”) The search only encompassed “Printed Books” (one of several searchable formats) so as to avoid double-counting books available in other formats. This means that books released only as e-books are omitted. To date, however, few scholars on this list pen books that are published solely as e-books. “Out of print” and forthcoming but as-yet-unreleased volumes were excluded, as were reports, commissioned studies, and special editions of magazines or journals. We only include books written in English, as relevant books which appear in other languages are almost always translations of an English text. This measure reflects the conviction that the visibility, packaging, and permanence of books allows them to play an outsized role in influencing policy and practice. Book points were capped at 20. (This search was conducted on Dec. 15.)

Highest Amazon Ranking: This reflects the scholar’s highest-ranked book on Amazon. The highest-ranked book was subtracted from 400,000, and the result was divided by 20,000 to yield a maximum score of 20. (In other words, a scholar’s best book had to rank in Amazon’s top 400,000 to earn points.) The nature of Amazon’s ranking algorithm means that this score can be volatile and favors more recent sales. The result is an imperfect measure but one that conveys real information about whether a scholar has penned a book that is influencing contemporary discussion of education policy and practice. (This search was conducted on Dec. 16.)

Syllabus Points: This seeks to measure a scholar’s long-term academic impact on what is being read by today’s college and university students. This metric was scored using OpenSyllabusProject.org, the most comprehensive extant database of syllabi. It houses over 6 million syllabi from across American, British, Canadian, and Australian universities. A search of the database was used to identify each scholar’s top-ranked text. The score reflects the number of times that text appeared on syllabi, with the tally then divided by 25. The score was capped at 10 points. (This search was conducted on Dec. 15-16.)

Newspaper Mentions: A LexisNexis search was used to determine the number of times a scholar was quoted or mentioned in U.S. newspapers. Again, searches used a scholar’s name and affiliation; diminutives and middle initials, if applicable, were included in the results. To avoid double counting the “Education Press” category, the scores do not include any mentions from Education Week, the Chronicle of Higher Education, or Inside Higher Ed. We grouped articles of “Moderate Similarity” (per LexisNexis) to avoid counting the same article multiple times. Points were capped at 30. (The search was conducted on Dec. 14.)

Education Press Mentions: This measures the total number of times the scholar was quoted or mentioned in Education Week, the Chronicle of Higher Education, or Inside Higher Education during 2020. Searches were conducted using each scholar’s first and last name. If applicable, searches included common diminutives; they were also conducted both with and without middle initials. For the Chronicle of Higher Education, mentions in the weekly book lists posts are excluded, as are mentions in the “Appointments, Resignations, and Deaths” column. The number of appearances in the Chronicle and Inside Higher Ed were averaged, and that tally was added to the number of times a scholar appeared in Education Week. (This was done to avoid overweighting the two higher education publications.) The resulting figure was multiplied by two, with total Ed Press points then capped at 30. (These searches were conducted on Dec. 14 and 16.)

Web Mentions: This reflects the number of times a scholar was referenced, quoted, or otherwise mentioned online in 2020. The intent is to use a “wisdom of crowds” metric to gauge a scholar’s influence on the public discourse. The search was conducted using Google. The search terms were each scholar’s name and university affiliation (e.g., “Bill Smith” and “Rutgers University”). Using affiliation served a dual purpose: It avoids confusion due to common names and increases the likelihood that mentions are related to university-affiliated activity. Variations of a scholar’s name (such as common diminutives and middle initials) were included in the search, if applicable. Web Mentions is a highly volatile metric, but it conveys useful information about a scholar’s presence and impact. Points were calculated by dividing total mentions by 30 and capped at 25. (This search was conducted on Dec. 16.)

Congressional Record Mentions: A simple name search in the Congressional Record for 2020 determined whether a scholar was referenced by a member of Congress. Qualifying scholars received 5 points. (This search was conducted on Dec. 14.)

Twitter Score: Followerwonk’s “Social Authority” score was used to calculate Twitter scores. Followerwonk scores each Twitter account on a scale of 0-100 based on the user’s retweet rate and other user-specific variables (such as follower count). While I’m highly ambivalent about the role played by social media, it’s indisputable that many public scholars exert significant influence via their social-media activity—and the lion’s share of this activity plays out on Twitter. To generate a point total, each score was divided by 10, yielding a maximum score of 10. (This search was conducted on Dec. 15-16.)

There are obviously lots of provisos when it comes to the Edu-Scholar results. Different disciplines approach books and articles differently. Senior scholars have had more opportunity to build a substantial body of work and influence (for what it’s worth, the results are unapologetically engineered to favor sustained accomplishment). And readers may care more for some categories than others. That’s all well and good. The intent is to spur discussion about the nature of constructive public influence: Who’s doing it, how much it matters, and how to gauge a scholar’s contribution.

A few notes regarding questions that arise every year:

  • There are some academics that dabble (quite successfully) in education but for whom education is only a sideline. They are not included in these rankings. For a scholar to be included, education must constitute a substantial slice of their scholarship. This helps ensure that the rankings serve as something of an apples-to-apples comparison.
  • Scholars sometimes change institutions in the course of a year. My policy is straightforward: For the categories where affiliation is used, searches are conducted using a scholar’s year-end affiliation. This avoids concerns about double-counting and reduces the burden on my overworked RAs. Scholars do get dinged a bit in the year they move. But that’s life.
  • Some eligible scholars wind up assuming deanships or serving as university provosts or presidents. The rule is that education school deans remain eligible, but that once a scholar moves to the larger role of university provost or president, they are no longer deemed eligible for the rankings.
  • It goes without saying that tomorrow’s list represents only a sliver of the nation’s education researchers. For those interested in scoring additional scholars, it’s a straightforward task to do so using the scoring rubric enumerated above. Indeed, the exercise was designed so that anyone can generate a comparable rating for a given scholar in a half hour or less.
  • This is an admittedly imperfect and evolving exercise. Questions and suggestions are always welcome. And, if scholars would like to have their names listed differently or have their discipline categorized differently, I’m happy to be as responsive as I can within the bounds of consistency.

Finally, a note of thanks: For the hard work of coordinating the Selection Committee, assembling the list of nominees, and crunching and double-checking the results for 200 scholars, I owe an immense debt of gratitude to my invaluable research assistants Hannah Warren, Hayley Sanon, and Tracey Schirra.

Related Tags:

The opinions expressed in Rick Hess Straight Up are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.


Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
6 Key Trends in Teaching and Learning
As we enter the third school year affected by the pandemic—and a return to the classroom for many—we come better prepared, but questions remain. How will the last year impact teaching and learning this school
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Webinar
How Data and Digital Curriculum Can Drive Personalized Instruction
As we return from an abnormal year, it’s an educator’s top priority to make sure the lessons learned under adversity positively impact students during the new school year. Digital curriculum has emerged from the pandemic
Content provided by Kiddom
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Law & Courts Federal Judge Denies Parents' Suit to Block Florida's Ban on School Mask Mandates
The parents argued that their children, due to health conditions, were at particular risk if any of their peers attend school without masks.
David Goodhue, Miami Herald
3 min read
Florida Governor Ron DeSantis speaks at the opening of a monoclonal antibody site in Pembroke Pines, Fla., on Aug. 18, 2021. The on-again, off-again ban imposed by Republican Gov. Ron DeSantis to prevent mandating masks for Florida school students is back in force. The 1st District Court of Appeal ruled Friday, Sept. 10, that a Tallahassee judge should not have lifted an automatic stay two days ago that halted enforcement of the mask mandate ban.
Florida Governor Ron DeSantis speaks at the opening of a monoclonal antibody site in Pembroke Pines, Fla., on Aug. 18, 2021. The on-again, off-again ban imposed by Republican Gov. Ron DeSantis to prevent mandating masks for Florida school students is back in force. The 1st District Court of Appeal ruled Friday, Sept. 10, that a Tallahassee judge should not have lifted an automatic stay two days ago that halted enforcement of the mask mandate ban.
Marta Lavandier/AP
Federal New Federal Team to Work on Puerto Rico School Improvement, Oversight
The Puerto Rico Education Sustainability Team will focus on creating better learning environments and improving financial management.
3 min read
Secretary of Education Miguel Cardona visits the Emilio Delgado School in Corozal on June 30, 2021 during a visit to Puerto Rico.
Secretary of Education Miguel Cardona visits the Emilio Delgado School in Corozal on June 30, 2021 during a visit to Puerto Rico.
Teresa Canino Rivera/GDA via AP
Law & Courts Texas Attorney General Sues More School Districts That Require Masks
The Texas attorney general's office anticipates filing more lawsuits against districts flouting the governor’s order. Will Dallas be next?
Talia Richman, The Dallas Morning News
4 min read
Texas Attorney General Ken Paxton speaks at the Austin Police Association in Austin, Texas, on Sept. 10, 2020.
Texas Attorney General Ken Paxton speaks at the Austin Police Association in Austin, Texas, on Sept. 10, 2020.
Jay Janner/Austin American-Statesman via AP
Policy & Politics Reported Essay What Is the Purpose of School?
We can’t build resilient schools until we agree what education's core role should be. And right now, we don’t agree.
8 min read
Conceptual illustration
Pep Montserrat for Education Week