School & District Management

UPDATED: Scholar Questions Induction Study Results

By Stephen Sawchuk — June 30, 2010 2 min read
  • Save to favorites
  • Print

The Institute of Education Science’s study on intensive teacher induction is getting a lot of buzz, and at least one scholar is worried that it may be giving an inaccurate picture of the effects of mentoring.

The study found positive effects in both reading and math that, relatively speaking, are quite large. As the New Teacher Center’s Liam Goldrick points out on his blog, the effect sizes for mentoring given in this report are larger than those produced in other large-scale randomized studies, such as one on Teach For America.

But Jonah Rockoff, an associate professor at Columbia University’s School of Business who’s studied induction programs in New York City, said to be cautious about the results. For one, he noted that despite these large achievement boosts, the data don’t show any other effects that would seem to confirm the results. Teachers didn’t report feeling any more prepared, for instance.

He also directed my attention to this factor: The researchers used “covariates” to control for the effects of teachers’ background characteristics and other factors on the data. Without such controls, the effect estimates drop and are no longer statistically significant for reading. You can find this on p. 95 of the report.

By Rockoff’s read, covariates shouldn’t be necessary in an experiment if the “treatment” and “control” group are really appropriate comparisons. One problem with large-scale education studies, he said, is that teachers often change classes, grade levels and subjects. In this case, as the population of teachers declined due to attrition and other movements, only a small population of teachers were tied to at least two years of student-achievement data. And if the declining group of teachers in the study resulted in a material difference in the composition of the treatment and control group, then that might skew the data.

“The big issue is whether treatment and control groups still look like one another, among the subset of teachers with multiple years of data,” Rockoff told me.

UPDATE: Steven Glazerman of Mathematica, one of the number guys behind the study, submits this response:

“We agree with Jonah Rockoff that one should be cautious about interpreting the test score impact findings. We sounded that note of caution in our report and in its executive summary because the results are not robust to alternative ways to estimate the program’s impact. However, the most credible estimates adjust for chance differences between student and teacher characteristics, including students’ achievement before they entered the study teacher’s classroom. These show a positive and significant impact on reading and math in two-year districts and no impact in one-year districts. Rockoff is correct that it is not necessary in a true experiment to adjust for these chance differences by including covariates. However, failing to do so would cause us to ignore data and rely on a less precise estimate. In this case especially, when the test score analysis sample is small, the precision gains are substantial: the standard error associated with our estimates decreases by 29% in a model with covariates compared to a model without covariates.”

A version of this news article first appeared in the Teacher Beat blog.

Commenting has been disabled on effective Sept. 8. Please visit our FAQ section for more details. To get in touch with us visit our contact page, follow us on social media, or submit a Letter to the Editor.


This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Teaching Webinar
What’s Next for Teaching and Learning? Key Trends for the New School Year
The past 18 months changed the face of education forever, leaving teachers, students, and families to adapt to unprecedented challenges in teaching and learning. As we enter the third school year affected by the pandemic—and
Content provided by Instructure
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Curriculum Webinar
How Data and Digital Curriculum Can Drive Personalized Instruction
As we return from an abnormal year, it’s an educator’s top priority to make sure the lessons learned under adversity positively impact students during the new school year. Digital curriculum has emerged from the pandemic
Content provided by Kiddom
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Equity & Diversity Webinar
Leadership for Racial Equity in Schools and Beyond
While the COVID-19 pandemic continues to reveal systemic racial disparities in educational opportunity, there are revelations to which we can and must respond. Through conscientious efforts, using an intentional focus on race, school leaders can
Content provided by Corwin

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management 10 Ways to Tackle Education's Urgent Challenges
As the school year gets underway, we ask hard questions about education’s biggest challenges and offer some solutions.
2 min read
Conceptual Image of schools preparing for the pandemic
Pep Montserrat for Education Week
School & District Management Reported Essay Principals Need Social-Emotional Support, Too
By overlooking the well-being of their school leaders, districts could limit how much their schools can flourish.
7 min read
Conceptual Illustration
Pep Montserrat for Education Week
School & District Management From Our Research Center Educator Stress, Anti-Racism, and Pandemic Response: How You're Feeling
A new nationally representative survey offers key takeaways from teachers, principals, and district leaders.
EdWeek Research Center
1 min read
2021 BI COVER no text DATA crop
Pep Montserrat for Education Week
School & District Management Download 8 Tips for Building a Digital Learning Plan That Conquers Chaos
Craft flexible strategies, encourage experimentation with new instructional models, and regularly solicit feedback.
1 min read
onsr edtech tips