Special Report
Student Well-Being

Researchers Play Catch-Up in Gauging Beyond-School Effects

By Sarah D. Sparks — April 01, 2011 7 min read
  • Save to favorites
  • Print

Emerging research shows the science school-age children learn in informal settings—from museums and clubs to online communities and television shows—can have a big impact on their lives. Yet the open format and distinct structures of informal science make it next to impossible for researchers to evaluate the quality of those experiences in the same way they can gauge formal schooling.

School assessments generally focus on cognitive measures, such as what a student knows and can demonstrate about particular content. In contrast, informal learning is dominated by noncognitive measures such as motivation, interest, and identity, according to Larry E. Suter, the National Science Foundation’s program director for informal science education. Moreover, traditional “gold standard” research methods such as randomized controlled trials can be detrimental to activities that base their strength on people’s choosing to participate, rather than being assigned.

Such research raises the risk, Mr. Suter said, that “if you touch it, you’re going to kill that thing you’re trying to study.”

That has led to some creative research alternatives. Alan J. Friedman, a former director and chief executive officer of the New York Hall of Science and the editor of the NSF’s 2008 framework for evaluating informal science education, recalls judging the effectiveness of an astronomy exhibit by the number of visitors who chose an astronomy poster over a different prize. Barbara N. Flagg, the director of the Multimedia Research consultant group in Bellport, N.Y., said she has used smudged museum walls indicating where visitors have touched exhibits, changes in Google and Amazon search terms over time, and phone interviews with parents and children.

“If you’re comparing this to other education research, you’ve got to turn back the clock 30 or 40 years,” said Kevin J. Crowley, the director of the University of Pittsburgh’s Center for Learning in Out-of-School Environments and an associate professor of education and psychology. “We are just now in the Wild West frontier, and people are just starting to gear up the longitudinal studies on how this will coalesce into a coherent narrative of how people learn science.”

Now, the tools being born of that creativity show the potential to link children’s education across a lifetime of different experiences, and in the process uncover more of how and what children learn than has ever been measured in a school test alone. Children, after all, spend more than 80 percent of their waking hours outside the classroom.

“The research has evolved,” Mr. Crowley said. “In the past, the great flaw of the informal-learning science was we looked at it in situations; we weren’t really looking at a lifelong trajectory in science learning. For the first time, we’re asking questions about how learning and participation are moving across place and across time.”

“Brenda’s” school performance, for instance, didn’t really show her scientific achievements or level of interest in the subject. Teachers reported the first-generation Haitian daughter of a single mother in Seattle had little interest in science and consistently “failed to engage” with the chemical-mixing tasks in her school lab.

Yet because researchers at the University of Washington, in Seattle, had been observing the 4th grader across formal and informal settings for more than 2,000 hours, they knew that school didn’t tell Brenda’s whole story. Not only did she regularly measure and mix chemicals and record the results for her perfume-making hobby, but she also had told the researchers she was considering becoming a chemist when she grew up.

“School science underrepresents her developing expertise,” Philip Bell, an associate professor of learning sciences at the university and the director of ethnographic and design-based research at the Everyday Science and Technology Group there, said during a recent lecture. “Just in terms of how people learn, our literatures don’t do justice to the varied pathways that people take through their experiences to make progress on things they care about.”

Vital Links

Learning Science in Informal Environments: People, Places, and Pursuits (2009)
The National Research Council of the National Academies, edited by Philip Bell, Bruce Lewenstein, Andrew W. Shouse, and Michael A. Feder
This landmark study by the National Academies’ Committee on Learning Science in Informal Environments documented evidence that children and adults do learn science outside of direct school instruction. Both designed science settings such as zoos or museums and spontaneous settings such as a walk in the park can help children understand science. The book laid the foundation for a more evidence-based approach to informal science education.

Surrounded by Science: Learning Science in Informal Environments (2010)
National Academies, Marilyn Fenichel and Heidi A. Schweingruber
A follow-up to the National Academies’ Learning Science in Informal Environments, this practitioner-focused guide provides case studies, framework tools, and other examples of how high-quality informal science education programs and exhibits can look.

Framework for Evaluating Impacts of Informal Science Education Projects (2008)
National Science Foundation, edited by Alan J. Friedman
Based on an NSF workshop on informal science education, this framework lays out the criteria for measuring informal science, based on a participant’s awareness, knowledge, or understanding of a science topic; engagement or interest in science; attitude toward science or careers in the field; changes in scientific behavior such as inquiry; and the improvement of specific skills related to science, such as experimenting or data analysis.

Measuring the Impact of a Science Center on Its Community (2011)
Journal of Research in Science Teaching, John H. Falk and Mark D. Needham
By studying Los Angeles residents and museum-goers before and a decade after a massive overhaul of the city’s California Science Center, researchers showed that the museum had increased the public’s understanding of and interest in the science covered in the new exhibits. The researchers used public understanding of homeostasis as a conceptual marker to track improved scientific understanding.

SOURCE: Education Week

Interest vs. Grades

That’s a dangerous disconnect, experts say, because mounting evidence shows that early engagement, even through informal pathways, eventually can lead to careers in the STEM fields of science, technology, engineering, and mathematics more surely than top grades in school.

In a 2006 study published in the journal Science, Robert H. Tai, an associate professor at the University of Virginia’s Curry School of Education, in Charlottesville, tracked thousands of students via the National Educational Longitudinal Study. He found that students who had only average grades in middle school but expressed interest in science were two to three times more likely to earn bachelor’s degrees in a science or engineering field 12 years later than high-achieving students who did not voice interest.

The landmark 2009 study that Mr. Bell co-wrote, “Learning Science in Informal Environments,” for the National Research Council, synthesized a vast array of research, including Mr. Tai’s, to conclude that “abundant evidence” exists that informal experiences can effectively support learning. What’s more, it found that those free-choice, low-stakes pathways can engage children from groups underrepresented in science fields: students in poverty, those from ethnic and language minorities, and women.

Still, while the number of research proposals submitted to the NSF about informal education has doubled in the past five years, from 300 to 600 annually, Mr. Suter said, the vast majority have been and continue to be from practitioners evaluating their individual programs, clubs, and exhibits, often using self-made assessments.

Better Alignment

In 2008, the Program in Education, Afterschool, and Resiliency at Harvard University and McLean Hospital in Boston found that of 64 widely used informal science evaluation tools, none met all five of the NSF’s five domains of informal learning: engagement and interest, attitude toward science and behavior, content knowledge, competence and reasoning, and career knowledge and acquisition.

Program founder Gil G. Noam and Rick Bonney, the director of program development and evaluation at the Cornell University Ornithology Lab, in Ithaca, N.Y., and the creator of several nationwide citizen-science projects, are working to change that. Mr. Noam has established the Assessment Tools in Informal Science database of existing tests. He said that he and Mr. Bonney are about a year away from using the tests to develop a single, unified assessment of informal science learning.

“Our big goal in all this is to bring into this field some proof that the informal science work makes a difference,” Mr. Noam said.

Likewise, Mr. Crowley’s organization, CAISE, has gathered five years’ worth of evaluation reports from NSF informal science projects—many never published—and has started to cull best practices and other lessons learned from across the many types of informal learning.

“Only recently has the learning theory caught up with those hunches, and now we’re at a place where we really can test design hypotheses based on what we know about the science of learning,” he said.

Ultimately, John H. Falk and Lynn D. Dierking, both professors in free-choice learning at Oregon State University, in Corvallis, hope the field will move toward more “synergy” studies, like Mr. Bell’s longitudinal work, that can explore how people learn science across the formal and informal experiences of their lives.

The Oregon State researchers now are monitoring how children in Portland, Ore., come to learn scientific concepts across the city’s many formal and informal science offerings during a four-year period. Portland provides a perfect microcosm for science learning, Mr. Falk said, because it has one science center, zoo, and public-broadcasting station and a few school districts that will allow the researchers to track individual students over time.

“In order to see real systemic improvement, you have to have a whole greater than the sum of its parts, and now the whole is less than the sum of its parts because there’s no alignment among these pieces,” Mr. Falk said. “We need to develop ways to better integrate practitioners across formal and informal education. We don’t understand how people navigate, make sense of, and make use of the resources in their community.”

This story was underwritten by a grant from the Noyce Foundation, at www.noycefdn.org.
A version of this article appeared in the April 06, 2011 edition of Education Week as Researchers Playing Catch-Up in Gauging Beyond-School Effects


Student Well-Being Webinar After-School Learning Top Priority: Academics or Fun?
Join our expert panel to discuss how after-school programs and schools can work together to help students recover from pandemic-related learning loss.
Budget & Finance Webinar Leverage New Funding Sources with Data-Informed Practices
Address the whole child using data-informed practices, gain valuable insights, and learn strategies that can benefit your district.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Classroom Technology Webinar
ChatGPT & Education: 8 Ways AI Improves Student Outcomes
Revolutionize student success! Don't miss our expert-led webinar demonstrating practical ways AI tools will elevate learning experiences.
Content provided by Inzata

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Student Well-Being Chaplains Could Work as School Counselors Under Bill Passed in Texas
Critics see the measure as a continuation of the erosion of the concept of separating church from state.
3 min read
This June 1, 2021, file photo shows the State Capitol in Austin, Texas.
This June 1, 2021, file photo shows the State Capitol in Austin, Texas.
Eric Gay/AP
Student Well-Being 4 States Consider Mandating Fentanyl Prevention Education in Schools
Oregon is poised to adopt the legislation, but drug education in schools is often weak or underemphasized.
4 min read
Photograph of Fentanyl opioid narcotic teaching awareness tools sitting on a definition page
Bojan Vujicic/iStock/Getty<br/>
Student Well-Being The U.S. Surgeon General's Warning About Social Media and What It Means for Schools
Schools have been ringing alarm bells over social media and kids' mental health. Now their cause is getting a major boost.
6 min read
Conceptual image of a young person engaged in social media.
Student Well-Being Opinion What Teachers Get Wrong About Creativity
Because of the priorities school systems set, teachers often stifle students' creativity without even realizing it.
Teresa Amabile
4 min read
Images shows a stylized artistic landscape with soothing colors.