Federal

States Wrestle With How to Evaluate Tutoring

By Catherine Gewertz — March 22, 2005 5 min read
  • Save to favorites
  • Print

States must evaluate the effectiveness of the free tutoring being provided to children under the federal No Child Left Behind Act. But a lack of resources might force them to compromise on the rigor of those evaluations, a new report says.

The policy brief outlines factors that states might want to consider in deciding whether providers of “supplemental educational services” are fulfilling their promises to raise student achievement. It acknowledges that states “may face a trade-off” between wanting rigorous evaluations and having the time, money, and staff to conduct them.

“Evaluating Supplemental Service Providers: Suggested Strategies for States” is available from tutorsforkids.org.

Released last week by the Supplemental Educational Services Quality Center, a federally funded project of the American Institutes for Research, based in Washington, the paper aims to guide states through the thorny process of designing such evaluations.

The advice comes as states are wrestling with the issue, since the 3-year-old federal law requires them to evaluate tutors after two years and to remove them from their lists of approved providers if they are not improving achievement. Providers themselves also are debating the fairest ways for states to assess their track records.

Inaccurate appraisals risk allowing ineffective companies to use public money to work with struggling youngsters, or unjustly excluding providers from a state marketplace when they have fulfilled their promises.

The stakes will only rise, observers note, as more schools must offer tutoring to children from low-income families if those schools have failed to make adequate academic progress three years in a row.

The number of students receiving such tutoring more than doubled between the 2002-03 and 2003-04 school years. Yet only 11 percent, on average, of those eligible are receiving tutoring. (“NCLB Choice Option Going Untapped, But Tutoring Picking Up,” March 16, 2005.)

Not There Yet

By anecdotal accounts, most states are not far along in designing evaluations. A recent informal inquiry by the American Federation of Teachers found only eight ready to roll. Even as states venture forth, however, many experts question their capacity to design and implement high-quality evaluations.

Ayeola Fortune, the director of extended learning and development opportunities for the Council of Chief State School Officers, in Washington, said most states are not yet confident that they can accurately measure the quality of tutoring services. Few states have the infrastructure or staff to monitor providers, or obtain enough data, she said.

See Also

See the accompanying item,

Chart: Grading Tutoring Programs

“Some of the more rigorous types of evaluation are going to be prohibitive for a lot of states, given what the capacity level is,” Ms. Fortune said. “But states are motivated to find an answer.”

Michael D. Casserly, the executive director of the Council of the Great City Schools, a Washington advocacy group for urban districts, said districts would welcome “more aggressive” state scrutiny of tutoring providers because there is no proof now that outside providers are serving students any better than the districts themselves.

The crucial issue, to many educators, is whether evaluations will be able to isolate and measure the effect of the tutoring.

“Even with the most burdensome and useful [evaluation], it doesn’t really get to the issue of what is the value added of these providers,” said Beth Antunez, an assistant director of educational issues for the AFT.

Steven M. Ross, the director of the Center for Research in Educational Policy at the University of Memphis, who wrote the policy brief in conjunction with the Supplemental Educational Services Quality Center, acknowledged that, even using a control group, “the best we have is suggestive evidence” that the extra help explains gains in achievement.

Some tutoring providers have suggested forming an accreditation system to evaluate their performance. Jeffrey H. Cohen, the president of Baltimore-based Catapult Learning, said many providers are wary that states will measure programs with varying goals by the same yardstick, yielding an unfair result.

Data Gathering

Providers also worry that if states adopt a common test, programs’ content will evolve to reflect that test and chase diversity from the marketplace, he said.

By many accounts, Louisiana is among the farthest along in designing the data-gathering basis for SES evaluations. The state has developed a data system for all of its after-school programs that can communicate with its data system for all other student information, said Donna Nola-Ganey, the assistant superintendent in the state education department’s office of school and community support.

Providers enter data on enrollment, daily attendance, and test scores along with their invoices for payment. When crossed with the other state data—including which students receive tutoring or other services—Louisiana will be able to compare the state test scores of children in tutoring programs with those who are not, Ms. Nola-Ganey said.

The state will also add parent and teacher comments to its evaluation, she said.

The paper suggests that states evaluate providers in three areas: effectiveness, including measures of student achievement; customer satisfaction, including input from parents, teachers, and school staff members; and service delivery, which asks whether providers offered what they promised in such areas as instructor qualifications, number of tutoring hours, and group size.

The center on supplemental-services quality encourages states and districts to collaborate in obtaining and sharing information needed for evaluations. It also suggests that each state consider creating a database into which all the tutoring-related data could be entered.

But the brief acknowledges that some data might not be readily available from schools or districts, and that some kinds of information-gathering are time-consuming or expensive, such as hiring translators to interview parents who don’t speak English. It also cautions that bias could color the inquiry, such as when teachers serve as instructors in tutoring programs, or districts themselves provide the programs.

States could use state-mandated tests to measure gains if yearly results were available, but couldn’t attribute any gains to tutoring unless they examined results for a control group of children who didn’t receive tutoring, the paper notes.

Use of providers’ own before-and-after tests could save time and money, but would risk bias because such tests are designed to measure the providers’ own programs and are more likely to show gains, it adds.

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Teaching Profession Webinar
Professional Wellness Strategies to Enhance Student Learning and Live Your Best Life
Reduce educator burnout with research-affirmed daily routines and strategies that enhance achievement of educators and students alike. 
Content provided by Solution Tree
English-Language Learners Webinar The Science of Reading and Multilingual Learners: What Educators Need to Know
Join experts in reading science and multilingual literacy to discuss what the latest research means for multilingual learners in classrooms adopting a science of reading-based approach.
School & District Management K-12 Essentials Forum Get a Strong Start to the New School Year
Get insights and actions from Education Week journalists and expert guests on how to start the new school year on strong footing.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Voters Want Republicans and Democrats to Talk About Learning Recovery, Not Culture Wars
A recent Democrats for Education Reform poll shows a disconnect between political candidates and voters on education issues.
4 min read
Image of voting and party lines.
TheaDesign/iStock/Getty
Federal Use Your 'Teacher Voice,' Jill Biden Urges in a Push for Political Activism
Voting in the midterms is a critical step educators can take to bolster democracy, the first lady and other labor leaders told teachers.
5 min read
First Lady Jill Biden speaks during the American Federation of Teachers convention, Friday, July 15, 2022, in Boston.
First lady Jill Biden speaks during the American Federation of Teachers convention in Boston.
Michael Dwyer/AP
Federal Federal Initiative Leverages COVID Aid to Expand After-School, Summer Learning
The Education Department's Engage Every Student effort includes partnerships with civic organizations and professional groups.
3 min read
Education Secretary Miguel Cardona speaks at an event on June 2, 2022, at the Department of Education in Washington.
U.S. Secretary of Education Secretary Miguel Cardona speaks at an event at the Department of Education in Washington in June. The department has announced a push for expanded access to after-school and summer learning programs.
Jacquelyn Martin/AP
Federal Restraint and Seclusion, and Disability Rights: Ed. Department Has Work to Do, Audit Finds
The Government Accountability Office releases a checklist of how the U.S. Department of Education is performing on a list of priorities.
4 min read
Flags decorate a space outside the office of the Education Secretary at the Education Department in Washington on Aug. 9, 2017.
Flags decorate a space outside the office of the Education Secretary at the U.S. Department of Education. The Government Accountability Office has released recommended priorities for the Education Department that target special education rights.
Jacquelyn Martin/AP