Chat Transcript: Quality Counts at 10: A Decade of Standards-Based Education
Quality Counts at 10: A Decade of Standards-Based Education
Jan. 25, 2006
Christopher Swanson, director of the EPE Research Center, discusses the Center’s recently released tenth edition of Quality Counts, “Quality Counts at 10: A Decade of Standards-Based Education.”
Melissa McCabe (Moderator):
Good afternoon and welcome to the third in a series of online chats following the release of this year’s Quality Counts report, “Quality Counts at 10: A Decade of Standards-Based Education.” Today we’re joined by Christopher Swanson, director of the EPE Research Center, which conducts annual policy surveys, collects data, and performs analyses that appear appear in the annual Quality Counts reports. Chris will discuss the report’s findings about student achievement trends and the relationship between standards-based reform and student performance.
Question from Lois Hetland, Research Associate, Project Zero, Harvard Graduate School of Education:
Is there any intention to measure the validity of these findings as they correlate with valued qualities of education beyond simple achievement? Do high scores on tests correlate with getting into better colleges, doing better while attending them, getting better jobs, being a more active citizen, or achieving higher satisfaction with life?
Christopher Swanson:
It’s no secret that educational researchers, policymakers, educators, parents, pretty much anyone interested in understanding how public schools are doing, spend a lot of time and effort looking at achievement scores. And by this we usually mean scores on standardized achievement tests. There are some good reasons for this.
One is that if we want to compare how one school in a state is performing relative to another school, we should do that on apples-to-apples terms. (The same consideration would also apply if we wanted to compare one state to another.) This is where the “standardized” part of standardized achievement test scores comes into play. These kinds of tests allow us to make valid comparisons from school to school. By contrast, the grades students receive in their classes (another performance measure that has important implications for individual students locally) are difficult to compare from place to place because expectations for student performance, content of classes taught, and standards for grading are much more variable across schools.
All that being said, results standardized achievement tests are useful to pay attention to because they also correlate closely with other student outcomes we may be interested in understanding better. Students with higher scores on standardized achievement test tend to earn higher grades in their classes, attend school more regularly, have less frequent discplinary problems, are more likely to graduate from high school and go to college, and so forth and so on. We don’t examine these connections specifically in Quality Counts, but there is a wealth of research on the way achievement test performance relates to these other outcomes.
Not every individual student who scores well on standardized tests will fit this profiles of success. Conversely, some students who do not perform well on those tests may work hard and do well in school and life more generally. (Afterall, it’s always good to remember that a youth’s life does not start end at the schoolhouse door.) But this is the general pattern that emerges from decades of research.
Question from BILL XANDER, RETIRED HS ENGLISH TEACHER, MONTGOMERY HIGH:
I believe that standards are necessary and helpful but I haven’t found any reliable way to measure them..Isn’t there some way to evaluate student achievement besides standardized testing?
Christopher Swanson:
Along the lines of the prior question, I think it’s good to recognize that standardized achievement tests serve some purposes well. In particular, making comparisons across schools or from state to state.
But any type of test has limitations. For examples, we can only put so many items on a test before it gets too long to administer to students. So it can be difficult to focus on all of the content in a particular subject area - math, science, etc., that we might view as being important. Likewise, a particular school or community may place a special value or importance on particular skills or knowledge that is less-highly valued by a neighboring community or a state as a whole. That is not to say that one point-of-view is right or wrong, just that priorities and expectations for education can vary and that it is hard to accomodate all values in standardized testing.
There is no perfect remedy, of course. But it is important to approach the question of student performance and success from multiple points of view so that we get a well-rounded perspective on a student. Some students excell in particular areas, their peers may have other strenghts. It is always good to recognize that. More importantly, it may help us to tailor instruction and learning strategies to best work with a particular student’s strengths and weaknesses.
Question from John Davis, professional developer, Western Pennsylvania Writing Project:
Many teachers complain that standards-focused and assessment-focused instruction pushes aside many valuable things in teaching. What do you think the standards movement has “pushed aside”?
Christopher Swanson:
This issue has come up a lot in recent years. In part this is sparked by attention to No Child Left Behind. But it is important to recognize that this is a concern with a longer history than the federal law. Many states were moving in the direction of more extensive testing and accountability systems while NCLB was still a glimmer in the eye of policymakers.
One thing we do know is that educators pay attention to what is tested and what stakes are attached to. This has positive aspects like focusing attention on important knowledge and skills. (Although the definition of “important skills” is something that is often negotiated and debated.) But there are also some potential downsides like “teaching to the test” and narrowing of curriculum.
There has been at least anecdotal evidence that high-stakes testing in certain areas (like math and English/language arts) may put the squeeze on instructional time devoted to other subjects. For example, some schools struggling in language arts may build extended class time into their schedule for English classes by shortening another class (art, social studies, gym, etc.).
While this is probably going on in some places, it is difficult to know how widespread this pattern might be or what the educational impacts on students might be.
Educators have a limited number of hours per day and days per year to teach students. At some point, choices have to be made about how best to spend seat time. The challenge is best knowing how to allocate time and resources to particular educational activities. Unfortunately there is no perfect formula for this which works for all schools or students.
Question from :
Could you give your take on the apparent negative relationship between teacher quality and results?
Christopher Swanson:
This is a finding from Quality Counts that has received a considerable amount of attention since we released the report. We also issued a separate research report (called “Making the Connection”) that goes into more detail about this analysis, which you can find online at www.edweek.org/qc06
In this analysis we examined the relationship between student achievement gains over the past decade or so and state implementaiton of policies related to standards-based reform. We found that states that had been more active in the areas of academic content standards, aligned assessments, and accountability measures, tended to have greater gains in achievement over time. We found this for math and reading at graded 4 and 8. But we didn’t find the same positive relationship for policies related to teacher quality.
I should say that we did not look at the actual characteristics of teachers here - just state policies INTENDED to improve teacher quality. So we’re not saying that teachers don’t matter. In fact there is plenty of research that confirms that teachers are one of the most important factors in delivering a quality education.
What our findings seem to suggest is that the policy levers states have been using for the past decade do not seem to be especially effective at producing gains in student achievement. There are many reasons why this might be the case but the focus of our study was on state-level policy levers. So a full examination was beyond the scope of our work here.
In some ways the findings raises more questions than it answers. But we think it does raise important questions. So we hope other researchers will conduct further work in this area.
Question from Norish L. Adams; Director, National Board Resource Center; Florida A&M University:
Accomplished National Board Certified Teachers(NBCTs)are reaching a “critical mass” (15-20 in one school) in schools across the state of Florida. Research is finding that students of NBCTs perform better than other students. Nationally, what role do you see for NBCTs in significantly improving student achievement_especially in high needs schools?
Christopher Swanson:
Again, following on the previous question, I think there is much we still don’t know about how to improve the quality and effectiveness of the teaching force. I cannot weigh in specifically on the effectiveness of this initiative. But I can say that it is vitally important to continue examing issues like the connection between increased student learning, and particular initiatives (like NBCT) and the concentration of highly-skilled teacher leaders in a school (the critical mass issue) are great subjects for research. Teaching effectively is a very complicated thing, some might say an art form. So the more we can understand about the intricacies of the student-teacher dynamic and the ways in which teachers work together to foster learning, the better able we will be to design reform strategies and policies that will have far-reaching effects on school improvement.
Question from Lisa Trygg, Program Manager, Stupski Foundation:
With regard to high school graduation data for urban districts, are recent calculations by district, using the Cummulative Promotion Index, readily available? What are the cautions in comparing CPI urban district data across state lines?
Christopher Swanson:
In this year’s Quality Counts, we report state graduation rates using the CPI calculation method. We look at both rates in the most recent year with data available and changes in graduation rates over the past decade. One of the reasons we used the CPI method is that we calculate rates in the same way using the same database across all of the states. This makes the information comparable from state-to-state. Had we, for example, used the statistics that the states report themselves we could not have made valid state-to-state comparisions (because the states calculate their graduation rates in different, sometimes inconsistent, ways).
Quality Counts does not report district-level graduation rates. But if you are interested in the topic of graduation and high school reform I would advise you to stay tuned. This June, Education Week and the EPE Research Center will be publishing the first of four annual special reports focusing on these issues. This project will include more local data on graduation rates. So, there will be much more to come on this topic.
Question from Joan Reinhart, Parent and taxpayer:
As there will always be wealthier and poorer school districts, schools with more resources or fewer resources, when one can’t establish parity in the education itself, how can one measure be used to evaluate disparate educational experiences? In the end, won’t some children benefit from standardization while others will be punished through the stigma of poor performance resulting from substandard educational opportnities?
Christopher Swanson:
I think it’s fair to say that fiscal inequities are a very widespread feature of most public education systems in this country, today. In part that is a result of the way public schools, historically, have been funded - largely through local tax dollars (of which there tend to be more in wealthier communities). But this is not to say that it will always be this way. In fact, the equity and adequacy of resources are a huge topic in a number of states right now (due to court cases and other factors).
Schools from poorer communities may have particular challenges to face that wealthier ones don’t, and may have fewer educational dollars to deal with these challenges. That’s not true in all cases, of course. But this does not mean that a standards-based strategy for educational improvement will fail to serve disadvantaged students. But if we are going to hold schools to higher expectations and standards, it is important that the resources and supports educators need to educate students to high standards are also available.
I think we are still very much in the process of figuring out how to marry high standards, equitable education, and adequate education.
Question from Laura Shapiro, Director of Education, The Center for Social and Emotional Education, NYC:
Our organization is working with schools to assess and improve the school climate in systemic, sustained ways. Our goal is to improve learning and the well being of all members of the school community. In you view, in what ways has the Standard’s movement impacted school climate? How have you assessed this impact and how might any negative impact be addressed? Thank you
Christopher Swanson:
Quality Counts has tracked indicators on school climate for a decade now. However, our analysis of policy effects on state student achievement gains did not look at climate specifically.
“School Climate” is something that is a little bit in the eye of the beholder. Our Quality Counts indicators in this area, for example, cover a lot of ground. In many ways I think the best way to think of initiatives to improve school climate is as facilitators of a standards-based approach. For example, efforts to reduce violence in schools may faciliate an orderly classroom and make it easier for a teacher to deliver quality instruction to high standards.
Researchers often focus a lot of effort in understanding the final outcomes like achievement scores and much less effort understanding the process of how high achivement comes about. But climate is important as an end in itself because this is the environment in which millions of students spend a good chunk of their waking hours. In addition, school climate factors may be particuarly vital to reforming schools as facilitators of learning. So more research on understanding climate from a process perspective could be very valuable.
Question from Rick Caporale, District One Representative, Beaufort (SC) County Board of Education:
What positive effects has NCLB had on student achievement?
Christopher Swanson:
I should begin by saying that Quality Counts does not attempt to examine the effectiveness of NCLB. It’s not a referrendum on the federal law.
But the kinds of policies we have been tracking for a decade now (many of which predate NCLB) can help us to better understand the kinds of standards-based educational principles that NCLB draws upon. In particular, we find that over the past decade states with stronger policy implementation in the areas of academic content standards, aligned assessments and accountability have seen larger gains in student achievement. These kinds of strategies are similar to some requirements of the federal law. So there may be some indirect lessons to be gained more generally about standards-based reform strategies (a category that could be argued to include NCLB). But many of these state policies were in place long before NCLB came on to the scene. So it doesn’t really tell us anything directly about No Child Left Behind.
Question from Meg Caddeau, Woodrow Wilson National Fellowship Foundation:
Which three or four states do you believe have achieved the greatest success thus far with standards-based education and reform at the high school level?
Christopher Swanson:
Quality Counts includes an extensive analysis of achivement data from the National Assessment of Educational Progress (NAEP), which was performed by ETS. This analysis tells us a great deal about progress at the elementary and middle school levels. The report includes special case studies of some states that have been particularly strong performers at these levels over the past decade. These include Delaware, New York, Texas, Massachusetts, and North Carolina.
But unfortunately, NAEP does not currently include a high school grade in its state assessment program. This is a big gap in our understanding about high school performance, just at a time when there has been so much attention to reforming our high schools.
Question from Amy Svirsky, Data Coordinator, Capital Region BOCES:
Do you see value added (growth) analysis joining the conversation about NCLB, assessment and accountability?
Christopher Swanson:
Certainly there has been much discussion lately about the potential use of growth models or other types of value added methodologies in the context of NCLB accountability. So the simple answer to your question is: “Yes.”
But this begs the further question of whether this is a good thing. That is a much more difficult question to answer. I think there both are upsides and downsides to adoption of a value-added approach. And the potential benefits and detriments are not always apparent until we really dig deeply into the issue. So I hope as this discussion evolves it will be a careful, thoughtful one.
Question from Bob Frangione, Graduate Student:
What has the effect of standards-based education been on the at-risk student population? Are these students being drawn back into the schools or are they giving up in the face of a higher challenge?
Christopher Swanson:
In looking at student achievement patterns over the past decade, we are finding some of the strongest gains being made by historically disadvantaged groups. For example, in Quality Counts we show that the nation as a whole has made large strides in 4th grade math when measured by NAEP - on the order of (roughly) two grade levels between 1992 and 2005. By comparison for Latino and African American students, gains have been around 2.5 to almost 3 grade levels (again, roughly). So, at-risk students are certainly not being left behind when we look at the national picture in math.
Reading, however, is a somewhat different story. Here we are seeing much less growth overall. But like math, some of the stronger gains are being made by at-risk students.
Question from Kristin Reedy, Director, Northeast Regional Resource Center:
What does your data show about subgroup performance, including the subgroup of students with disabiltes under NCLB. Do you have state by state disaggregated data by subgoup compared to the overall school population?
Christopher Swanson:
Quality Counts did not report information specifically for students with disabilities. However NAEP (the source of our achievement data) does collect data on disability status, so you might be able to find more information through NAEP reports issues by U.S. Department of Education, which conducts NAEP. The Department of Education also has a very powerful publicly accessible online data analysis tool that can provide you with information on issues of subgroup performance more generally.
Question from Dorothy Singleton,Vice President, Advocates for Communites and Rural Education,Arkansas:
Has standard based education reform caused an increase in dropout rates in the US?
Christopher Swanson:
There have been a number of studies in recent years looking at the relationship between high-stakes accountability policies and dropout rates. There are a variety of methodological challenges involved in trying to answer this question (which is actually a very complex one, despite its apparent simplicity). Some studies do a better job than others. But in the end, results from different researchers point in different directions. So I think we still do not have a definitive answer.
Question from Suzanne Forman, Senior Associate, Teaching that Makes Sense:
Is it the standards that have an effect on student performance, or quality teaching?
Christopher Swanson:
The answer is probably both. Standards and quality teaching can work hand-in-hand. Clear, specific standards can give teachers an important resource to help guide their practice. And standards can probably be taught most effectively by a high-quality teacher. While we have a fair amount of resarch on standards and research on teacher quality (how ever one might choose to define that loaded term), we may not have enough that puts these two pieces together. It would be nice to see more of that more nuanced perspective on the standards-teacher quality issue.
Question from Rhonda Stone, parent advocate, Read Right Systems:
A single spreadsheet with average NAEP 4th and 8th grade reading and math scores for all 50 states, early 1990s through 2005, shows that more individual state scores DECLINED in reading in one particular year than in any of the previous years in reading or math. That year was 2005. From a statistical point of view, what does it mean when reforms in one subject area produce steady gains state-by-state while reforms in another produce a significant percentage of decline?
Christopher Swanson:
Across our analysis we find a number of places where results looks much different for math (typically very strong)and reading (typically less strong).
There are two major factors that may be coming into play here. First, math is a more school-based subject where students may be better able to pick up literacy skills outside school. This could make math instruction and achievement more amenable (in general) to the influence of policies or other initiatives rooted in the school.
The other issue that may be coming into play is the history of the standards movements in math and language arts. Math was an area where there was very early and strong leadership around a standardards-based agenda. There was also relatively little controversy. By contrast, English/language arts reforms did not take off as early and were fairly quickly enmeshed in politically-charged debates over how to define the “cannon” in literature and related issues (think “culture wars”). So the policy and reform movement in math has been stronger and more consistently pursued.
Melissa McCabe (Moderator):
I’m afraid we’ve run out of time. Chris, thanks for participating. Next Wednesday, from 3 to 4 EST, we’ll host another chat with Marshall S. Smith, the program director for education at the William and Flora Hewlett Foundation, about what is next for standards-based education reform.