Federal

States Wrestle With How to Evaluate Tutoring

By Catherine Gewertz — March 22, 2005 5 min read
  • Save to favorites
  • Print

States must evaluate the effectiveness of the free tutoring being provided to children under the federal No Child Left Behind Act. But a lack of resources might force them to compromise on the rigor of those evaluations, a new report says.

The policy brief outlines factors that states might want to consider in deciding whether providers of “supplemental educational services” are fulfilling their promises to raise student achievement. It acknowledges that states “may face a trade-off” between wanting rigorous evaluations and having the time, money, and staff to conduct them.

“Evaluating Supplemental Service Providers: Suggested Strategies for States” is available from tutorsforkids.org.

Released last week by the Supplemental Educational Services Quality Center, a federally funded project of the American Institutes for Research, based in Washington, the paper aims to guide states through the thorny process of designing such evaluations.

The advice comes as states are wrestling with the issue, since the 3-year-old federal law requires them to evaluate tutors after two years and to remove them from their lists of approved providers if they are not improving achievement. Providers themselves also are debating the fairest ways for states to assess their track records.

Inaccurate appraisals risk allowing ineffective companies to use public money to work with struggling youngsters, or unjustly excluding providers from a state marketplace when they have fulfilled their promises.

The stakes will only rise, observers note, as more schools must offer tutoring to children from low-income families if those schools have failed to make adequate academic progress three years in a row.

The number of students receiving such tutoring more than doubled between the 2002-03 and 2003-04 school years. Yet only 11 percent, on average, of those eligible are receiving tutoring. (“NCLB Choice Option Going Untapped, But Tutoring Picking Up,” March 16, 2005.)

Not There Yet

By anecdotal accounts, most states are not far along in designing evaluations. A recent informal inquiry by the American Federation of Teachers found only eight ready to roll. Even as states venture forth, however, many experts question their capacity to design and implement high-quality evaluations.

Ayeola Fortune, the director of extended learning and development opportunities for the Council of Chief State School Officers, in Washington, said most states are not yet confident that they can accurately measure the quality of tutoring services. Few states have the infrastructure or staff to monitor providers, or obtain enough data, she said.

See Also

See the accompanying item,

Chart: Grading Tutoring Programs

“Some of the more rigorous types of evaluation are going to be prohibitive for a lot of states, given what the capacity level is,” Ms. Fortune said. “But states are motivated to find an answer.”

Michael D. Casserly, the executive director of the Council of the Great City Schools, a Washington advocacy group for urban districts, said districts would welcome “more aggressive” state scrutiny of tutoring providers because there is no proof now that outside providers are serving students any better than the districts themselves.

The crucial issue, to many educators, is whether evaluations will be able to isolate and measure the effect of the tutoring.

“Even with the most burdensome and useful [evaluation], it doesn’t really get to the issue of what is the value added of these providers,” said Beth Antunez, an assistant director of educational issues for the AFT.

Steven M. Ross, the director of the Center for Research in Educational Policy at the University of Memphis, who wrote the policy brief in conjunction with the Supplemental Educational Services Quality Center, acknowledged that, even using a control group, “the best we have is suggestive evidence” that the extra help explains gains in achievement.

Some tutoring providers have suggested forming an accreditation system to evaluate their performance. Jeffrey H. Cohen, the president of Baltimore-based Catapult Learning, said many providers are wary that states will measure programs with varying goals by the same yardstick, yielding an unfair result.

Data Gathering

Providers also worry that if states adopt a common test, programs’ content will evolve to reflect that test and chase diversity from the marketplace, he said.

By many accounts, Louisiana is among the farthest along in designing the data-gathering basis for SES evaluations. The state has developed a data system for all of its after-school programs that can communicate with its data system for all other student information, said Donna Nola-Ganey, the assistant superintendent in the state education department’s office of school and community support.

Providers enter data on enrollment, daily attendance, and test scores along with their invoices for payment. When crossed with the other state data—including which students receive tutoring or other services—Louisiana will be able to compare the state test scores of children in tutoring programs with those who are not, Ms. Nola-Ganey said.

The state will also add parent and teacher comments to its evaluation, she said.

The paper suggests that states evaluate providers in three areas: effectiveness, including measures of student achievement; customer satisfaction, including input from parents, teachers, and school staff members; and service delivery, which asks whether providers offered what they promised in such areas as instructor qualifications, number of tutoring hours, and group size.

The center on supplemental-services quality encourages states and districts to collaborate in obtaining and sharing information needed for evaluations. It also suggests that each state consider creating a database into which all the tutoring-related data could be entered.

But the brief acknowledges that some data might not be readily available from schools or districts, and that some kinds of information-gathering are time-consuming or expensive, such as hiring translators to interview parents who don’t speak English. It also cautions that bias could color the inquiry, such as when teachers serve as instructors in tutoring programs, or districts themselves provide the programs.

States could use state-mandated tests to measure gains if yearly results were available, but couldn’t attribute any gains to tutoring unless they examined results for a control group of children who didn’t receive tutoring, the paper notes.

Use of providers’ own before-and-after tests could save time and money, but would risk bias because such tests are designed to measure the providers’ own programs and are more likely to show gains, it adds.

Events

Reading & Literacy Webinar How Background Knowledge Fits Into the ‘Science of Reading’ 
Join our webinar to learn research-backed strategies for enhancing reading comprehension and building cultural responsiveness in the classroom.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Webinar
Innovative Strategies for Data & Assessments
Join our webinar to learn strategies for actionable instruction using assessment & analysis.
Content provided by Edulastic
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
IT Management Webinar
Build an Effective EdTech Stack for Your Students
Join respected experts to learn about best practices and possibilities for tech-enabled redesign and inclusivity.
Content provided by Class

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Student Activists Push Congress for Action on Climate Change Education
Student activists collaborated to write a pair of U.S. House resolutions on climate change.
4 min read
Composite image of school building and climate change protestors.
Illustration by F. Sheehan/Education Week (Images: iStock/Getty and E+)
Federal House GOP Passes Parents' Rights Bill in Clash Over Schools
The legislation seeks to give parents a role in what's taught in public schools, but it has little chance in the Democrat-run Senate.
4 min read
Speaker of the House Kevin McCarthy, of Calif., right, speaks about the proposed legislation dubbed the "Parents Bill of Rights," on March 1, 2023, next to Rep. Elise Stefanik, R-N.Y., on Capitol Hill in Washington.
Speaker of the House Kevin McCarthy, of Calif., right, speaks about the proposed legislation dubbed the "Parents Bill of Rights," on March 1, 2023, next to Rep. Elise Stefanik, R-N.Y., on Capitol Hill in Washington.
Jacquelyn Martin/AP
Federal How Schools Fare in Biden's Proposed Budget
President Joe Biden released his fiscal 2024 budget, which calls for more money for preschool, Title I, and special education.
7 min read
President Joe Biden speaks about his 2024 budget proposal at the Finishing Trades Institute, Thursday, March 9, 2023, in Philadelphia.
President Joe Biden speaks about his 2024 budget proposal at the Finishing Trades Institute, Thursday, March 9, 2023, in Philadelphia.
Evan Vucci/AP
Federal The Push for a $60K Base Teacher Salary Gains Steam as Bernie Sanders Signs On
Sanders' legislation complements a proposal in the House and comes as state lawmakers from both parties prioritize legislation to boost teacher pay.
6 min read
Sen. Bernie Sanders, I-Vt., outlines his priorities during an interview in his Capitol Hill office, Feb. 7, 2023.
Sen. Bernie Sanders, I-Vt., outlines his priorities during an interview in his Capitol Hill office, Feb. 7, 2023.
J. Scott Applewhite/AP