Includes updates and/or revisions.
The statistician who pioneered the use of “value added” research techniques for documenting students’ year-to-year academic progress is disputing a critique of his approach that was published recently in a prominent academic journal.
In the article, which appeared in the March issue of Educational Researcher, Audrey Amrein-Beardsley, an assistant professor at Arizona State University in Phoenix, takes aim at the Education Value-Added Assessment System, or EVAAS. The EVAAS is an offshoot of the Tennessee Value-Added Assessment System developed by William L. Sanders, which is widely credited with helping to launch nationwide interest in value-added approaches for measuring students’ academic progress.
Ms. Amrein-Beardsley contends that the EVAAS is opaque, is possibly flawed, and has not been subjected to enough external review to merit its widespread use in school systems.
“No one is looking inside the model to see if the assumptions being made within it are true,” Ms. Amrein-Beardsley said in an interview. “The bottom line is to open this up for debate and not have these things be adopted blindly.”
‘Factual Errors’ Alleged
A recently posted response to Audrey Amrein-Beardsley’s critique is now available online. The response is written by William L. Sanders and S. Paul Wright.
But Mr. Sanders, for his part, contends that Ms. Amrein-Beardsley’s report is “rife with factual errors,” and he points to three studies in which researchers have duplicated and tested his model. Among the most recent was a 2007 study, published in the peer-reviewed Electronic Journal of Statistics, in which a team of researchers from the Santa Monica, Calif.-based RAND Corp. compared the Sanders method with other value-added techniques.
Now working in private industry as a senior manager for value-added assessment and research at the Cary, N.C.-based SAS Institute, Mr. Sanders has never shared the computational algorithms for the EVAAS. But he and a colleague described the model in a 1997 paper, which the RAND researchers and others have used to create their own software mimicking Mr. Sanders’ model.
Ms. Amrein-Beardsley also raises technical concerns with the EVAAS, some of which echo cautions that other experts are voicing about value-added modeling in general. (“Scrutiny Heightens for ‘Value Added’ Research Methods,” May 7, 2008.)
She notes, for instance, that the EVAAS does not take into account demographic and socioeconomic differences between students.
“Family income, ethnicity, ability, and other background variables unquestionably affect levels of student achievement and the progress that students make from year to year,” she writes.
Mr. Sanders, though, has long maintained that such statistical adjustments are not needed because the EVAAS compares students’ year-to-year academic growth with their own progress from previous years. “A kid is coming from the same home this year as he was the year before,” he said.
To illustrate her case, Ms. Amrein-Beardsley reanalyzed data from the appendix of a controversial 2005 study of North Carolina teachers that Mr. Sanders was recruited to do for the National Board for Professional Teaching Standards, an Arlington, Va.-based group that awards national credentials to teachers who meet the requirements of a process based on documentation of their performance.
While Mr. Sanders concluded that board-certified teachers did no better on average than those who had failed to earn certification and those who had never tried, Ms. Amrein-Beardsley said the data showed a positive student-achievement effect. She said that, on average, students of board-certified teachers outperformed those of teachers with only regular certification by three-quarters of a month’s worth of growth in mathematics and one week in reading.
Mr. Sanders said his conclusions stemmed in part from another finding in his study: The variation in student-achievement levels for teachers within the NBPTS-certified group was greater than it was for teachers across different categories.
Joseph A. Aguerrebere Jr., the board’s chief executive officer, said most studies of board-certified teachers, such as the one by Ms. Amrein-Beardsley, also find that students of those teachers outperform their peers.
