Assessment Chat

The Role of International Tests in U.S. Education

The mediocre performance of U.S. students on international exams, such as the PISA, TIMSS, and PIRLS, has prompted concern among American policymakers in recent years.

Chat: The Role of International Tests in U.S. Education

Thursday, April 23, 2-3 p.m. Eastern time; read the transcript.

The mediocre performance of U.S. students on international exams, such as the PISA, TIMSS, and PIRLS, has prompted concern among American policymakers in recent years. Some say the results should prompt a re-examination of U.S. school policies and further study of the best education practices used by other, higher-performing countries.

In this chat, our panelists examined the proper role of international tests in shaping U.S. education policy. What do these tests reveal about our performance? Do they represent the kinds of academic benchmarks that will prepare U.S. students for success in a global society, and what lessons can we draw from the results?

Guests: Gary Phillips, vice president and chief scientist at the American Institutes for Research

Hal Salzman, professor of public policy and a senior faculty fellow at the John J. Heldrich Center for Workforce Development at Rutgers University

Kathleen Kennedy Manzo, associate editor at Education Week, moderated this chat.

Related Coverage:
• A Nation at Risk: 25 Years Later

Live Chat: The Role of International Tests in U.S. Education

(04/23/2009)

Close

1:30	Web Person: Jennifer: Today’s chat on the role of international tests in U.S. education is now open for questions, so please start submitting them. Our moderator and guests will here at 2 p.m. to begin answering your questions. Thanks for joining us.

1:58	Kathleen Kennedy Manzo: Hi everyone. Thanks for joining us. Let’s begin the chat. Why don’t our guests, Hal Salzman and Gary Phillips, introduce themselves first, and then they’ll start answering your questions.

1:59

Hal Salzman:

This is Hal Salzman – I’m Professor of Public Policy and sociologist at the Bloustein School of Public Policy and the Heldrich Center for Workforce Development at Rutgers University. With colleagues, I have been examining the science and engineering workforce “demand” and “supply”, which led us to look closely at the education performance of U.S. students, finding that our best students are actually doing quite well, contrary to the widely-cited proclamations of poor performance.

2:00

Gary W. Phillips:

I am Gary W. Phillips the Vice President & Chief Scientist at the American Institutes for Research (AIR). I direct a team of psychometricians and statisticians on large scale assessment projects. I recently published several reports that involve statistically linking NAEP to TIMSS. The first report compared U.S. state-NAEP results to TIMSS in math and science (Chance Favors the Prepared Mind, 2007), the second compared U.S. School District results to TIMSS (Counting on the Future, 2008) and a new report is about to be released that compares states and school district to TIMSS using letter grades A, B, C, D and F. (International Benchmarks in Mathematics, 2009). Prior to coming to AIR, I was the Acting Commissioner at the National Center for Education Statistics.

2:01	Kathleen Kennedy Manzo: Thank you to both of you for sharing your insights and your expertise on this topic. Can each of you briefly characterize your views on the value of international comparison tests, and what the results do and do not tell us about how our schools are doing?

2:03

Hal Salzman:

There is certainly value in comparing schools and education systems to learn from others. However, the conclusions often drawn from these results go far beyond what the actual data support. U.S. academic performance at the top is actually quite good comparatively, as the most recent tests show.

2:03	Gary W. Phillips: International assessments give the United States a broader context within which to understand our own student learning and teaching practices. They do not provide a silver bullet but they add to the knowledge base that helps us decide on best practice.

2:03	Kathleen Kennedy Manzo: Gary, can you answer this question from Olga?

2:03	[Comment From Olga Amaral] Is there one common factor that has been found to be most influential in contributing to the success of countries with highest achievements?

2:05	Gary W. Phillips: I think we do not have a final answer to this question. But it seems to me to be related to how well education is respected and valued within the society. Those countries that care the most about education achieve the highest.

2:06	Kathleen Kennedy Manzo: Here’s one Hal from Nick

2:06

[Comment From Nick]
Shouldn’t we be more worried about U.S. graduation rates, and performance in local schools, rather then worrying about American scores on international tests? Why is it that international test are being looked at to help shape our education policies? Shouldn’t we be looking towards better training for teachers and focus on curriculum rather then focusing on changing policies?

2:07	Hal Salzman: I would agree and we’ve argued that we have more to learn from high performing districts and states (e.g., Massachusetts and Minnesota) than from Finland -- if only beause of greater transferrability to US context.

2:07	Gary W. Phillips: All of these factors are important to look at. We should be looking at graduation rates, achievement as well as better taecher traing and improved curriculum.

2:07	Kathleen Kennedy Manzo: Gary there are a couple of questions about the comparisons themselves, when the tests are given, the comparability of groups. I’ll post one or two for you.

2:07	[Comment From Robert D. Ferrari] How are the comparison groups of various countries selected? Are they representative groups based on the same set of criteria?

2:08

[Comment From Stuart Hobbs]
My question concerns differences in educational goals between the US and many other countries that makes the comparisons on international tests (which I find interesting and a good idea) somewhat problematic. Specifically, in the US we are trying, more or less, to educate everyone to the same level, while in other places by our equivalent of middle school students have been separated on academic or vocational tracks. If that is so, won’t that mean that while we can certainly improve on our test scores, we will always be running behind other countries if their test results include only the academic track and ours include everyone?

2:08	Hal Salzman: The testing groups try to use comparable samples but the reality of a large, complicated study is that samples often are not comparable. In the PISA, sampling differences may account for a significant portion of the outcome differences.

2:09	Gary W. Phillips: Each sample in each country is a scientifically representative sample. An international sampling referee makes sure that each country complies with sampling requirements.

2:10

Hal Salzman: Gary, I would agree that they try to do so, and make a good effort at it, but they don’t always succeed. There are substantial differences in the final sample and the U.S. sample in particular has been a problem in several of the tests. In our papers we discuss in detail the impact of sampling problems.

2:11	Gary W. Phillips: I disagree with Dr Salzmans conclussion. If there is one thing that we can trust in these studies it is that the sampling is done well. The minor sampling problems that due emerge due no make much difference.

2:11	Kathleen Kennedy Manzo: Can either of you answer this question from Kris?

2:11	[Comment From Kris Palfrey] What countries are achieving the highest scores and what examination boards are they using?

2:13	Kathleen Kennedy Manzo: In this week’s Education Week we have a series of articles on other countries that are doing well and some of the factors that might be contributing to their high performance on international assessments. www.edweek.org

2:13	Gary W. Phillips: The highest achieving countries are Chinese Taipei, South Korea, Singapore, Hong Kong and Japan and Finland. Each has a different exam system.

2:15	Kathleen Kennedy Manzo: Hal, can you address this question from Jerry?

2:15

[Comment From Gerald W. Bracey]
Given the Japan scores among the highest nations in the world but has seen 20 years of economic recession and stagnation, and given that highscoring Iceland became an overnight economic basket case with a national debt equal to 850% of its Gross Domestic Product, and given that middle-scoring United States has long been ranked first in global competitiveness by both the Institute for Management Development and the World Economic Forum, why should anyone pay any attention to these test scores?

2:17

Hal Salzman: There are two questions here --- is there anything to be learned from educational testing and is economic performance the primary outcome that we should be interested in. There is a qualified yes to the first and, as you make clear in the question, economic performance is due to many other factors. It is, quite obviously, not reasonable to focus on education as the primary driver of economic performance.

2:18

Kathleen Kennedy Manzo:

Just some background on these tests and the results, from my colleague Sean Cavanagh:

American students have fared much better on one international test, the TIMSS, than on another PISA. There’s a lot of speculation about why. The TIMSS, like the domestic NAEP test, measures school-based curriculum. The PISA measures “yield,” or the skills that students have picked up in and out of school, and their ability to apply them to real-world contexts.

For PISA results: http://nces.ed.gov/pubs2008/2008016.pdf

And TIMSS: http://nces.ed.gov/timss/results07.asp

2:18	Kathleen Kennedy Manzo: Gary, you have done some studies linking the scores for international tests on a scale with NAEP results. What did you find?

2:20	Hal Salzman: As additional follow up to Jerry’s earlier question, it is also worth noting that the U.S. tops the scale on reading and other subjects. It is arguable that only math and science are the most important subjects.

2:20	[Comment From Milad] Are there alignment studies between TIMSS and international tests?

2:20	Gary W. Phillips: This observation is an artifact of how the two surveys calculate the international average to which we compare the United States. PISA uses the 30 OECD countries so our standaing relative to them is lower. That is why we appear to do less well on PISA than on TIMSS.

2:21	Kathleen Kennedy Manzo: Anne is referring to efforts by states to rewrite state standards using international benchmarks. Can one of you answer her question?

2:21	[Comment From Anne] 2. We are hoping our state will rewrite its standards, using international standards as benchmarks. What other factors should we consider?

2:21	Gary W. Phillips: There are content aligment studies between all the international assessments and they can be found by going to NCES web.

2:23	Gary W. Phillips: There “is” a way to internationally benchmark state achievement standards using PISA, TIMSS and PIRLS. I am doing a workshop on this with state testing directors the first week in May.

2:24

Kathleen Kennedy Manzo: This is a hot topic right now.

Two influential organizations, the National Governors Association and the Council of Chief State School Officers, in December called for setting common state standards in reading and math and benchmarking standards in high-performing countries.( See Ed Week’s story on a report issued by the two organizations, at http://www.edweek.org/ew/articles/2008/12/18/16nga.h28.html)

And just last week, representatives of 41 states convened at a conference in Chicago to explore those issues. The event was co-sponsored by the NGA and CCSSO. Efforts to promote national standards have been promoted, and fizzled, in the past. Will this new effort overcome that fate, and what role would international benchmarks play, if any?

http://www.edweek.org/ew/articles/2009/04/16/29standards.h28.html

Just yesterday the House Education and Labor Committee announced a hearing, set for April 29, on “to examine how states can better prepare their students to compete in a global economy by using internationally benchmarked common standards.”

2:24	Kathleen Kennedy Manzo: Hal, what do you think these efforts to identify international benchmarks and align state standards accordingly?

2:26

Hal Salzman:

This, of course, raises the question about what we think is important for competing globally. Presumably it is performance after they leave school and not clear that the U.S. is at a disadvantage. We still lead the world in innovation, R&D, etc. Perhaps there is something about our education system that leads to the US being the world’s leaders in science, technology, innovation, entrepreneurship. Moreover, it is our universities that seem to produce the world’s best graduates.

2:26	Kathleen Kennedy Manzo: Gary, Patti has a question about participation on these tests, particularly for India and China.

2:26	[Comment From Patti] Is it possible to know how U.S. students compare academically with students from mainland Chinese and India--because those countries don’t participate in the PISA or TIMMS?

2:27	Hal Salzman: If one factors in the 30-50% illiteracy or near-literacy rates of those countries, their averages would be pretty close ot the bottom.

2:28	Gary W. Phillips: We do not know how the U.S. compares to these two countries. However, I believe PISA plans to include both countries (or at least a portion of both countries) in their next assessment.

2:29	Kathleen Kennedy Manzo: This next question is for Gary, regarding your earlier response on high-performing countries

2:29	[Comment From Robert Pondiscio] I’m intrgued by your response that “those countries that care the most about education achieve the highest.” How are you defining caring? Economically? Or through some manner of value judgement?

2:29	Kathleen Kennedy Manzo: Hal, can you answer Karen’s question?

2:29	[Comment From Karen] In the Education Week article, Mr. Salzman, you mention that many U.S. students fall out of the math and science pipeline. Is that a problem for these other high-performing countries as well?

2:31	Kathleen Kennedy Manzo: I have a better link for the article Karen is referring to, as well as all of Education Week’s recent coverage of this issue over the last year, in commemoration of the 25th anniversary of the A Nation at Risk Report http://www.edweek.org/go/nar

2:31	Gary W. Phillips: If you look inside the cultures of the high achieving countries you will see that teachers are well respected, teaching is a competitive job, parents are involved with school and students take school seriously.

2:31	Hal Salzman: We don’t know because we haven’t tracked enfollments in those countries. We do find many engineers entering management, for example, in Japan and Germany, so there is some career mobility. Overall there tends to be more tracking by discipline.

2:32

Kathleen Kennedy Manzo:

Here’s some more background from Sean Cavanagh:

In a 2007 paper, Hal Salzman and Lindsay Lowell, of Georgetown University, argued that the United States K-12 system produces an ample number of students with skills necessary to compete in science and engineering fields, contrary to widespread belief. American students’ math and skills are equal and in some cases better than they were two decades ago, they conclude. The problem lies in graduate programs and science- and engineering-related companies, which are ineffective at recruiting talented students and convincing them to stay. http://www.edweek.org/ew/articles/2007/11/07/11report-b1.h27.html

2:33	[Comment From Louise Johnson] Would the formation and implementation a national curriculum have an affect upon international assessments? If so, what are your opinions?