Researchers Play Catch-Up in Gauging Beyond-School Effects
What's measured in the classroom—what students know and can do—differs from what's currently measured outside—such as motivation and interest
Emerging research shows the science school-age children learn in informal settings—from museums and clubs to online communities and television shows—can have a big impact on their lives. Yet the open format and distinct structures of informal science make it next to impossible for researchers to evaluate the quality of those experiences in the same way they can gauge formal schooling.
School assessments generally focus on cognitive measures, such as what a student knows and can demonstrate about particular content. In contrast, informal learning is dominated by noncognitive measures such as motivation, interest, and identity, according to Larry E. Suter, the National Science Foundation’s program director for informal science education. Moreover, traditional “gold standard” research methods such as randomized controlled trials can be detrimental to activities that base their strength on people’s choosing to participate, rather than being assigned.
Such research raises the risk, Mr. Suter said, that “if you touch it, you’re going to kill that thing you’re trying to study.”
That has led to some creative research alternatives. Alan J. Friedman, a former director and chief executive officer of the New York Hall of Science and the editor of the NSF’s 2008 framework for evaluating informal science education, recalls judging the effectiveness of an astronomy exhibit by the number of visitors who chose an astronomy poster over a different prize. Barbara N. Flagg, the director of the Multimedia Research consultant group in Bellport, N.Y., said she has used smudged museum walls indicating where visitors have touched exhibits, changes in Google and Amazon search terms over time, and phone interviews with parents and children.
“If you’re comparing this to other education research, you’ve got to turn back the clock 30 or 40 years,” said Kevin J. Crowley, the director of the University of Pittsburgh’s Center for Learning in Out-of-School Environments and an associate professor of education and psychology. “We are just now in the Wild West frontier, and people are just starting to gear up the longitudinal studies on how this will coalesce into a coherent narrative of how people learn science.”
Now, the tools being born of that creativity show the potential to link children’s education across a lifetime of different experiences, and in the process uncover more of how and what children learn than has ever been measured in a school test alone. Children, after all, spend more than 80 percent of their waking hours outside the classroom.
“The research has evolved,” Mr. Crowley said. “In the past, the great flaw of the informal-learning science was we looked at it in situations; we weren’t really looking at a lifelong trajectory in science learning. For the first time, we’re asking questions about how learning and participation are moving across place and across time.”
“Brenda’s” school performance, for instance, didn’t really show her scientific achievements or level of interest in the subject. Teachers reported the first-generation Haitian daughter of a single mother in Seattle had little interest in science and consistently “failed to engage” with the chemical-mixing tasks in her school lab.
Yet because researchers at the University of Washington, in Seattle, had been observing the 4th grader across formal and informal settings for more than 2,000 hours, they knew that school didn’t tell Brenda’s whole story. Not only did she regularly measure and mix chemicals and record the results for her perfume-making hobby, but she also had told the researchers she was considering becoming a chemist when she grew up.
“School science underrepresents her developing expertise,” Philip Bell, an associate professor of learning sciences at the university and the director of ethnographic and design-based research at the Everyday Science and Technology Group there, said during a recent lecture. “Just in terms of how people learn, our literatures don’t do justice to the varied pathways that people take through their experiences to make progress on things they care about.”
Learning Science in Informal Environments: People, Places, and Pursuits (2009)
The National Research Council of the National Academies, edited by Philip Bell, Bruce Lewenstein, Andrew W. Shouse, and Michael A. Feder
This landmark study by the National Academies’ Committee on Learning Science in Informal Environments documented evidence that children and adults do learn science outside of direct school instruction. Both designed science settings such as zoos or museums and spontaneous settings such as a walk in the park can help children understand science. The book laid the foundation for a more evidence-based approach to informal science education.
Surrounded by Science: Learning Science in Informal Environments (2010)
National Academies, Marilyn Fenichel and Heidi A. Schweingruber
A follow-up to the National Academies’ Learning Science in Informal Environments, this practitioner-focused guide provides case studies, framework tools, and other examples of how high-quality informal science education programs and exhibits can look.
Framework for Evaluating Impacts of Informal Science Education Projects (2008)
National Science Foundation, edited by Alan J. Friedman
Based on an NSF workshop on informal science education, this framework lays out the criteria for measuring informal science, based on a participant’s awareness, knowledge, or understanding of a science topic; engagement or interest in science; attitude toward science or careers in the ﬁeld; changes in scientific behavior such as inquiry; and the improvement of speciﬁc skills related to science, such as experimenting or data analysis.
Measuring the Impact of a Science Center on Its Community (2011)
Journal of Research in Science Teaching, John H. Falk and Mark D. Needham
By studying Los Angeles residents and museum-goers before and a decade after a massive overhaul of the city’s California Science Center, researchers showed that the museum had increased the public’s understanding of and interest in the science covered in the new exhibits. The researchers used public understanding of homeostasis as a conceptual marker to track improved scientiﬁc understanding.
Interest vs. Grades
That’s a dangerous disconnect, experts say, because mounting evidence shows that early engagement, even through informal pathways, eventually can lead to careers in the STEM fields of science, technology, engineering, and mathematics more surely than top grades in school.
In a 2006 study published in the journal Science, Robert H. Tai, an associate professor at the University of Virginia’s Curry School of Education, in Charlottesville, tracked thousands of students via the National Educational Longitudinal Study. He found that students who had only average grades in middle school but expressed interest in science were two to three times more likely to earn bachelor’s degrees in a science or engineering field 12 years later than high-achieving students who did not voice interest.
The landmark 2009 study that Mr. Bell co-wrote, “Learning Science in Informal Environments,” for the National Research Council, synthesized a vast array of research, including Mr. Tai’s, to conclude that “abundant evidence” exists that informal experiences can effectively support learning. What’s more, it found that those free-choice, low-stakes pathways can engage children from groups underrepresented in science fields: students in poverty, those from ethnic and language minorities, and women.
Still, while the number of research proposals submitted to the NSF about informal education has doubled in the past five years, from 300 to 600 annually, Mr. Suter said, the vast majority have been and continue to be from practitioners evaluating their individual programs, clubs, and exhibits, often using self-made assessments.
In 2008, the Program in Education, Afterschool, and Resiliency at Harvard University and McLean Hospital in Boston found that of 64 widely used informal science evaluation tools, none met all five of the NSF’s five domains of informal learning: engagement and interest, attitude toward science and behavior, content knowledge, competence and reasoning, and career knowledge and acquisition.
Program founder Gil G. Noam and Rick Bonney, the director of program development and evaluation at the Cornell University Ornithology Lab, in Ithaca, N.Y., and the creator of several nationwide citizen-science projects, are working to change that. Mr. Noam has established the Assessment Tools in Informal Science database of existing tests. He said that he and Mr. Bonney are about a year away from using the tests to develop a single, unified assessment of informal science learning.
“Our big goal in all this is to bring into this field some proof that the informal science work makes a difference,” Mr. Noam said.
Likewise, Mr. Crowley’s organization, CAISE, has gathered five years’ worth of evaluation reports from NSF informal science projects—many never published—and has started to cull best practices and other lessons learned from across the many types of informal learning.
“Only recently has the learning theory caught up with those hunches, and now we’re at a place where we really can test design hypotheses based on what we know about the science of learning,” he said.
Ultimately, John H. Falk and Lynn D. Dierking, both professors in free-choice learning at Oregon State University, in Corvallis, hope the field will move toward more “synergy” studies, like Mr. Bell’s longitudinal work, that can explore how people learn science across the formal and informal experiences of their lives.
The Oregon State researchers now are monitoring how children in Portland, Ore., come to learn scientific concepts across the city’s many formal and informal science offerings during a four-year period. Portland provides a perfect microcosm for science learning, Mr. Falk said, because it has one science center, zoo, and public-broadcasting station and a few school districts that will allow the researchers to track individual students over time.
“In order to see real systemic improvement, you have to have a whole greater than the sum of its parts, and now the whole is less than the sum of its parts because there’s no alignment among these pieces,” Mr. Falk said. “We need to develop ways to better integrate practitioners across formal and informal education. We don’t understand how people navigate, make sense of, and make use of the resources in their community.”
Vol. 30, Issue 27, Pages s6, s7Published in Print: April 6, 2011, as Researchers Playing Catch-Up in Gauging Beyond-School Effects