Opinion
School & District Management Opinion

Making Sense of School Improvement Program Evaluations (II): The Case of TEEM

January 05, 2008 3 min read
  • Save to favorites
  • Print

(Readers’ please note: The December 20 posting generated a great deal of email. A comment worth making is one that should be posted on the blog. Emails to me that are not prefaced with “not for publication” are subject to posting.)

On December 20 I posted a piece on Edvance’s review of the Texas Early Education Model. The bottom line of that work, which covered only the first two years of a four-year effort, was equivocal:

There was considerable variation both between and within communities with regards to student performance and teacher outcomes. For about half of the communities, students in the treatment groups (with TEEM) improved more than students in the control groups (without TEEM), and for the other half of the communities students in the control groups improved more than the students in the treatment groups on the student outcome measures. TEEM did lead to overall improvement for teachers, although there was considerable variation, with teachers in both control and treatment groups obtaining both positive and negative difference scores on the teacher outcome measure.

Staci Hupp of the Dallas Morning News translated this into “no proof that most children fared better in TEEM than in conventional preschool programs.” How should policymakers and taxpayers read the results? Like Hupp’s headline - “Landmark preschool program isn’t paying off”? And how should we think about school improvement program evaluation?
Many found the evaluation’s finding discouraging. Quite a few edbizbuzz reader despise the program and the provider - and let other readers know.

As someone with experience in the evaluation of education programs on a large scale, I found this part of the Edvance report intruiging:

“For about half of the communities, students in the treatment groups (with TEEM) improved more than students in the control groups (without TEEM), and for the other half of the communities students in the control groups improved more than the students in the treatment groups on the student outcome measures.”

What was different about the two groups of communities? The Edvance evaluation tells us nothing about this. But we know from other research (for example, see here, here and here) that outcomes relate to the quality of implementation and implementation relates to the quality of teacher and agency support. This also relates to improvements for teachers - it’s quite unreasonable to expect teachers who do not buy into a program to improve by measures designed by that program. If the communities with superior performance had higher levels of program implementation and higher levels of support, it would not be accurate to imply that the program wasn’t working. However, we might infer that the program is only likely to work where it’s wanted, so the idea that it should become a statewide preschool strategy is flawed.

The advocates of TEEM are probably shooting themselves in the foot by pushing for statewide implementation, because they are almost certainly assuring mediocre results “on average.” But opponents equally shortsighted, because it’s quite likely that teachers and district administrators who share a belief in TEEMs efficacy will use it to the benefit of higher student performance.

There’s nothing overly complicated about this logic.

If you really believe in a diet program and find it fits your life style, you are more likely to use it, and so lose weight. Maybe there’s a plan out there that will allow you to lose even more weight, but if you don’t like it you won’t use it. And if you don’t use it, you won’t lose weight.

School improvement is no different. The products and services are not pills; they are programs. If teachers don’t like them, if administrators won’t provide the support, their benefits are purely theoretical. Providers who want to demonstrate high levels of effectiveness should not be eagerly accepting clients who will merely impose their programs on teaching staffs. District administrators who think they can obtain advertised results by imposing a program on teachers are fools. Teachers who don’t protest the imposition of programs they will not implement faithfully are setting themselves up for failure.

It would be nice if more research would focus on this problem, because it lies at the core of program efficacy.

Related Tags:

The opinions expressed in edbizbuzz are strictly those of the author(s) and do not reflect the opinions or endorsement of Editorial Projects in Education, or any of its publications.

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Recruitment & Retention Webinar
Be the Change: Strategies to Make Year-Round Hiring Happen
Learn how to leverage actionable insights to diversify your recruiting efforts and successfully deploy a year-round recruiting plan.
Content provided by Frontline
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Equity & Diversity Webinar
Critical Ways Leaders Can Build a Culture of Belonging and Achievement
Explore innovative practices for using technology to build an environment of belonging and achievement for all staff and students.
Content provided by DreamBox Learning
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Professional Development Webinar
Strategies for Improving Student Outcomes with Teacher-Student Relationships
Explore strategies for strengthening teacher-student relationships and hear how districts are putting these methods into practice to support positive student outcomes.
Content provided by Panorama Education

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management Download A Visual Guide to Nonverbal Communication (Download)
Understanding nonverbal communication can help you improve interactions and get your message across.
1 min read
v42 8SR Nonverbal Communication Share Image
Gina Tomko/Education Week and Getty
School & District Management Ensure Your Staff Gets the Message: 3 Tips for School Leaders
School staff are inundated with information. Here's a few ways to ensure they will actually hear you.
3 min read
Image showing a female and male in business attire connecting speech bubble puzzle pieces.
iStock/Getty Images Plus
School & District Management Keep School Staff Motivated All Year Long: Advice From Principals
Here are some of the things—big and small—that school leaders do and say to keep teachers excited about the job.
13 min read
Teachers and faculty play a game of Kahoot! to get to know one another better during a Welcome Back training at CICS Bucktown on Monday, Aug. 15, 2022 in Chicago, Ill.
Teachers and faculty play a game to get to know one another better during a Welcome Back training at Chicago's CICS Bucktown in August.
Taylor Glascock for Education Week
School & District Management How District Leaders Can Make Social Media Work for Them
Two school district leaders with impressive followings share best practices for using social media.
3 min read
Two diverse educators with laptops sitting on an oversize cellphone with communication symbols and text bubbles on the phone and in the air around them.
Gina Tomko/Education Week and DigitalVision Vectors