Is the No Child Left Behind Act working?
President Bush says it is, pointing to student-achievement results from a single subsection of the National Assessment of Educational Progress and tentative Reading First data. But the evidence available to support his claim is questionable.
“Fourth graders are reading better,” the president said during a March 2 visit to a school in New Albany, Ind. “They’ve made more progress in five years than the previous 28 years combined.”
In mathematics, he said, elementary and middle school students “earned the highest scores in the history of the test.”
The data Mr. Bush cited at that event are from just the “long-term trend” NAEP in reading and math, researchers say. All available data, they add, show modest improvements that can’t be attributed to the 5-year-old law. Instead, progress in achievement is more likely a continuation of trends that predate the law.
“There’s not any evidence that shows anything has changed,” said Daniel M. Koretz, a professor of education at Harvard University’s graduate school of education.
Other researchers suggest that the standards and accountability system of the NCLB law is drawing attention to achievement gaps and other inequalities and is causing educators to change their practice. But it’s too early to say whether the federal law will result in achievement gains, they contend.
The law’s “mechanisms are just coming into play, and not enough time has passed to establish a trend,” said Adam Gamoran, a professor of sociology at the University of Wisconsin-Madison.
‘I’m Lobbying Congress’
Portraying the No Child Left Behind law as a success is a critical element in President Bush’s argument that Congress should renew it on schedule this year. The president signed the legislation, an overhaul of the Elementary and Secondary Education Act, with much fanfare in January 2002 and has cited it as his most important accomplishment in domestic policy.
“I’m not only speaking to you, I’m lobbying,” Mr. Bush said at the Silver Street Elementary School in New Albany earlier this month. “I’m lobbying Congress. I’m setting the stage for Congress to join me in the reauthorization of this important piece of legislation.”
Congress is laying the groundwork for reauthorizing the measure. This week, the Senate education committee held a hearing on the law’s teacher-quality requirements. Next week, the House and Senate education committees plan to hold a joint session on an overview of the law.
Rep. George Miller, D-Calif., and Sen. Edward M. Kennedy, D-Mass., the chairmen of the education committees and two of the architects of the bipartisan law, say they hope to renew it this year. But many observers expect the process will be delayed until next year or even after Mr. Bush leaves office in 2009.
At the New Albany school, Mr. Bush highlighted the gains on the national assessment’s long-term-trend tests in reading and mathematics. Secretary of Education Margaret Spellings pointed to the same NAEP data on the law’s fifth anniversary in January, and during several other recent speeches.
President Bush likes to cite the “long-term-trend” NAEP as proof that the No Child Left Behind Act is working. The gains are significant only for 9- and 13-year-olds in math and 9-year-olds in reading. What’s more, the gains fall into a five-year testing window, and only two of those years occurred after the law took effect.
*Click image to see the full chart.
Between 1999 and 2004, the reading scores of 9-year-olds climbed from 212 to 226 on the test’s 500-point scale. The gap between African-American and white students that age narrowed to 26 points in 2004, compared with 35 points five years earlier. The gap between Hispanic 9-year-olds and their non-Hispanic white peers tapered from 24 points to 21 points in that same time period.
On the “national” NAEP, meanwhile, researchers say the advances in math reflect a continuation of student-achievement progress since 1990. Fourth graders are dead-even with where they were in reading when the law took effect in 2002. The slight decline in 8th grade reading scores is not statistically significant.
*Click image to see the full chart.
Note: Lighter shading indicates years when testing accommodations weren’t allowed.
SOURCE: National Center for Education Statistics
On the math test, 9-year-olds’ scores rose by 9 points, and the gaps between Hispanics’ and African-Americans’ scores and whites’ scores narrowed slightly as well.
Although the results for 9-year-olds on the reading test are positive, researchers say they can’t be linked to the law. The testing window extends back to 1999—three years before President Bush signed the NCLB legislation into law and even before he was president.
“With some of the claims that Spellings has made, for most of the time period there was no NCLB, so she can’t really say [any improvement] is because of the law,” said Gerald W. Bracey, the author of Reading Educational Research: How to Avoid Getting Statistically Snookered,who runs a LISTSERV, or e-mail forum, tracking what Mr. Bracey calls the administration’s “disinformation.”
Mr. Bracey, a frequent critic of testing programs, points out that implementation of the law began in 2002, but didn’t start to fuel significant change in schools until the 2003-04 school year. “So I guess [the Bush administration] should be sharing some of the credit with the Clinton administration,” he said.
In math, the gains since 2002 are the extension of an upward trend that dates back more than 20 years, researchers say.
“They just pay attention to what happened after NCLB,” said Jaekyung Lee, an associate professor of education at the State University of New York at Buffalo. “Part of it is just a continuation of a trend from pre-NCLB.”
The administration appears to ignore other data that suggest the law has had little or no positive effect on achievement.
On a different NAEP exam, gains haven’t been as significant, Mr. Lee said. What is known as the “national” NAEP, as distinguished from the long-term-trend tests, shows 4th grade reading scores the same in 2005 as three years earlier, when the law was signed. Math scores rose 1 point between 2003 and 2005. While that increase was statistically significant, it was smaller than the 9-point gain between 2000 and 2003.
The scores on the “national” NAEP demonstrate that the NCLB law’s impact is incomplete, said Katherine McLane, the U.S. Department of Education’s press secretary.
“The secretary is the first to say we have more work to do,” Ms. McLane said in response to the criticisms. “That is one of the issues we have to look at in education.”
Regardless of whether NAEP scores go up or down, it’s almost impossible to link those changes to the NCLB law without a well-designed research study, said Mr. Koretz of Harvard. That would compare a group of students who were exposed to NCLB policies against one that hadn’t participated in the testing and accountability measures in the law.
Those are the types of studies that the Bush administration says must be presented as evidence to select reading materials for the Reading First program and to win approval for research grants from the department.
Also, scores in the upper grades on both versions of the national assessment are for the most part unchanged from before the law’s passage.
NAEP is given to a sampling of students nationwide. Scores on states’ own tests, however, are used to determine whether schools have made adequate yearly progress under the federal law. Mr. Gamoran of the University of Wisconsin said the debate over NAEP scores is probably irrelevant. Even in 2005, the law’s most significant policies weren’t fully phased in. Those include the requirements that all teachers be “highly qualified” and that all states annually assess math and reading achievement in grades 3-8 and once in high school, said Mr. Gamoran, the director of the university’s Wisconsin Center for Education Research.
‘Reading First’ Results
In addition to speeches citing the NAEP long-term-trend data, members of the Bush administration have lauded the success of the $1 billion-a-year Reading First program, the largest new initiative in the NCLB law.
In the administration’s blueprint for the reauthorization of the No Child Left Behind Act, unveiled in January, the Education Department described Reading First as “the largest, most focused, and most successful early-reading initiative ever undertaken in this country.”
Few disagree that it is the largest and most focused. The initiative, which requires that participating schools use “scientifically based” materials and assessments, includes more than 5,600 schools in 1,600 districts. An estimated 100,000 teachers have had some kind of professional development associated with the program, according to the blueprint.
But there is scant empirical evidence showing the program’s effect on student achievement. An independent interim study on Reading First implementation, released last year, included survey results from state officials. It showed that the program had led to significant increases in the time participating schools spent on reading instruction, as well as more substantive professional development and support for teachers, and the use of assessment data to inform instruction.
A later survey, conducted by the Center on Education Policy, a Washington-based research and advocacy group, indicated that states were generally pleased with the program, with most claiming some improvement in student scores on state tests.
President Bush’s blueprint includes preliminary results showing some gains in students’ reading fluency. “For the 2004-05 school year, students in Reading First schools demonstrated increases in reading achievement across all performance measures,” Education Department officials wrote in the blueprint.
“The percentage of 2nd grade students who met or exceeded proficiency in reading on Reading First outcome measures of fluency increased from 33 percent in 2003-04 to 39 percent in 2004-05 for economically disadvantaged students; from 27 to 32 percent for [limited-English proficient] students; from 34 to 37 percent for African-American students; from 30 to 39 percent for Hispanic students; and from 17 to 23 percent for students with disabilities,” the document adds.
Those gains, however, are based on a compilation of all test results in annual state reports for Reading First.
That compilation includes results from the DIBELS assessment, or Dynamic Indicators of Basic Early Literacy Skills, developed by researchers at the University of Oregon and used in more than 35 states to monitor student progress on fluency and other measures. But they also include results from a variety of other assessments, including the Iowa Test of Basic Skills and Terra Nova.
“The results show that more kids in the early grades are making great progress on learning the basic components of reading under Reading First,” Ms. McLane, the department’s press secretary, said of the data reported in the blueprint.
Although such an assemblage of test scores can provide a general view of student progress, some researchers question whether the compilation says much about reading proficiency.
“If the goal is just to see if students are improving, I think there is nothing wrong with using different tests as long as it is established that the tests are reliable and valid, and reasonably comparable,” Stephen D. Krashen, an education researcher and linguist at the University of Southern California, in Los Angeles, wrote in an e-mail. However, “many [researchers] feel that DIBELS is not valid.”
Critics of DIBELS cite the tendency of some educators to teach to the tests or give the measures too much weight in judging reading ability. They also question whether a test that gauges how many words a student can read accurately in a minute, as DIBELS does, is a valid indicator of their proficiency. (“National Clout of DIBELS Test Draws Scrutiny,” Sept. 28, 2005.)
According to Mr. Bracey, fluency—the ability to read a text accurately and quickly—is not a good indicator of reading mastery, which requires comprehension.
“Kids can be very fluent and not have a clue about what they just read,” he said.
Success of Standards
While most researchers say it’s too early to measure the NCLB law’s impact on achievement, many are beginning to see evidence that educators are changing their behavior as a result of both the federal law and policies that took root in the 1990s at the onset of the movement for higher standards and greater accountability in education.
“The big success of No Child Left Behind so far is to galvanize attention to the challenges we face, particularly the challenges of inequity,” Mr. Gamoran said.
But critics of the law question, in any case, the central place it gives to test scores. They say it puts too much emphasis on the negative consequences of failing to meet annual student-performance targets and glosses over the professional development and other interventions needed to improve struggling schools and get to the heart of elevating student achievement.
“What’s troublesome about it is the idea that you can eliminate [achievement] gaps by putting pressure on schools and nothing else,” said Gary A. Orfield, the director of the Civil Right Project at Harvard and the University of California, Los Angeles. “It’s making a bad situation worse.”
