Blog

Your Education Road Map

Politics K-12®

ESSA. Congress. State chiefs. School spending. Elections. Education Week reporters keep watch on education policy and politics in the nation’s capital and in the states. Read more from this blog.

Education

Interpreting i3 Scores: Good Luck

By Michele McNeil — August 06, 2010 3 min read
  • Save to favorites
  • Print

After announcing the 49 winning applicants for the $650 Investing in Innovation competition, the U.S. Department of Education has now put online the scores, judges’ comments, and more details about each project.

Trying to make sense of the numerical scores for the validation and development award winners is, at least for this blogger, an exercise in futility. And it’s all because of a statistical process called “standardization.”

For me, the quest to understand the i3 scoring system began with this question: Did Saint Vrain School District really have the best application of them all?

Because of the large number of applicants and judges in the validation and development categories, the department used standardization to make sure the scoring was done as fairly as possible. (For more about this, read the department’s explanation on page 2 of this FAQ document.) It’s important to note that for the 19 scale-up applicants, standardization was not used, and their scores are their raw scores. Also, for Race to the Top, raw scores were used and no adjustments were made for scoring anomalies, which did spark some questions.

As a case study in standardization—and in how meaningless the i3 judges’ actual raw scores really are—let’s look at Saint Vrain School District’s score sheet. This caught my attention because not only did this Colorado school district win a development grant, it had the highest score of all applicants, at 116.95. (That’s on a grading scale of 105; more on that later.)

This application and other winning ones had five judges: three subject-matter experts who zeroed in on the proposal and two research experts who evaluated the evidence presented. All attended mandatory department training sessions.

In looking at the judges’ scores, you’d have no idea this was a winning application, much less the highest-rated of them all. And the two who judged the evidence components seemed to raise legitimate questions about the research behind the district’s proposal, and how the district planned to evaluate its new program to help English-language learners by focusing on STEM, or science, technology, engineering, and math, subjects. (To be clear, this is not a critique of Saint Vrain, but more an exploration of the i3 score standardization process.)

One subject-matter judge gave Saint Vrain just 42 out of 80 points possible. Another one gave it 50 out of 80. And one judge gave it 75 out of 80. On the evidence side, the district got 10 of 25 points from one judge and 13 out of 25 points from another.

Other winning applications got higher, more consistent scores. And my guess is if you looked at the raw scores for applications that didn’t win, there would be some that were better than Saint Vrain’s.

So I asked the department last night: How in the world, with not-so-great raw scores like that, did Saint Vrain win and get the highest score?

Department officials told me it all boils down to standardization, which seeks to balance out hard graders and easy graders across applications so that an applicant isn’t penalized for getting assigned hard graders. For example, the two subject-matter judges who gave Saint Vrain those low scores must have been very, very hard graders, as evidence by the grades they gave on other applications as well. The same thing must have been true for the research-expert judges, who must also have been very tough graders.

But as one Politics K-12 reader who was also questioning Saint Vrain’s standardized score pointed out, how likely is it that a truly strong proposal would be marked that low by anyone, let alone two subject-matter judges and two research judges?

So, at least from my point of view, this makes the scores you see on the score sheets relatively meaningless. And even the comments may not be as meaningful as they could be because some of the tough graders were judged, as least by some statistical program, to be too tough, while other graders were judged to be too easy.

Moreover, it’s also confusing that the final scores listed by the department go above 105, which is the maximum amount of points available on the competition’s grading scale. Again, blame standardization.

I am by no means a statistical expert, so please weigh in on this issue in the comments section below. I’m especially interested in other noteworthy things you’ve spotted on the i3 score sheets.


Commenting has been disabled on edweek.org effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Webinar
What’s Next for Teaching and Learning? Key Trends for the New School Year
The past 18 months changed the face of education forever, leaving teachers, students, and families to adapt to unprecedented challenges in teaching and learning. As we enter the third school year affected by the pandemic—and
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Curriculum Webinar
How Data and Digital Curriculum Can Drive Personalized Instruction
As we return from an abnormal year, it’s an educator’s top priority to make sure the lessons learned under adversity positively impact students during the new school year. Digital curriculum has emerged from the pandemic
Content provided by Kiddom
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Education Judge's Temporary Order Allows Iowa Schools to Mandate Masks
A federal judge ordered the state to immediately halt enforcement of a law that prevents school boards from ordering masks to be worn.
4 min read
Iowa Gov. Kim Reynolds speaks to reporters following a news conference, Thursday, Aug. 19, 2021, in West Des Moines, Iowa. Reynolds lashed out at President Joe Biden Thursday after he ordered his education secretary to explore possible legal action against states that have blocked school mask mandates and other public health measures meant to protect students against COVID-19. Reynolds, a Republican, has signed a bill into law that prohibits school officials from requiring masks, raising concerns as delta variant virus cases climb across the state and schools resume classes soon. (AP Photo/Charlie Neibergall)
Education Hurricane Ida Deals New Blow to Louisiana Schools Struggling to Reopen
The opening of the school year offered teachers a chance to fully assess the pandemic's effects, only to have students forced out again.
8 min read
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021. Louisiana students, who were back in class after a year and a half of COVID-19 disruptions kept many of them at home, are now missing school again after Hurricane Ida. A quarter-million public school students statewide have no school to report to, though top educators are promising a return is, at most, weeks away, not months.
Six-year-old Mary-Louise Lacobon sits on a fallen tree beside the remnants of her family's home destroyed by Hurricane Ida, in Dulac, La., on Sept. 4, 2021.
John Locher/AP
Education Massachusetts National Guard to Help With Busing Students to School
250 guard personnel will be available to serve as drivers of school transport vans, as districts nationwide struggle to hire enough drivers.
1 min read
Massachusetts National Guard soldiers help with logistics in this Friday, April 17, 2020 file photo, at a food distribution site outside City Hall, in Chelsea, Mass. Mass. Gov. Charlie Baker on Monday, Sept. 13, 2021, activated the state's National Guard to help with busing students to school as districts across the country struggle to hire enough drivers.
Massachusetts National Guard soldiers help with logistics in this Friday, April 17, 2020 file photo, at a food distribution site outside City Hall, in Chelsea, Mass.
Michael Dwyer/AP
Education FDA: ‘Very, Very Hopeful’ COVID Shots Will Be Ready for Younger Kids This Year
Dr. Peter Marks said he is hopeful that COVID-19 vaccinations for 5- to 11-year-olds will be underway by year’s end. Maybe sooner.
4 min read
Dr. Peter Marks, director of the Center for Biologics Evaluation and Research in the Food and Drug Administration, testifies during a Senate health, education, labor, and pensions hearing to examine an update from federal officials on efforts to combat COVID-19 on Capitol Hill in Washington on May 11, 2021. On Friday, Sept. 10, 2021, Marks urged parents to be patient, saying the agency will rapidly evaluate vaccines for 5- to 11-year-olds as soon as it gets the needed data.
Dr. Peter Marks, director of the Center for Biologics Evaluation and Research in the Food and Drug Administration, testifies during a Senate health, education, labor, and pensions hearing to examine an update from federal officials on efforts to combat COVID-19 on Capitol Hill in Washington on May 11, 2021.
Jim Lo Scalzo/AP