Special Report
Student Well-Being

Researchers Play Catch-Up in Gauging Beyond-School Effects

By Sarah D. Sparks — April 01, 2011 7 min read
  • Save to favorites
  • Print

Emerging research shows the science school-age children learn in informal settings—from museums and clubs to online communities and television shows—can have a big impact on their lives. Yet the open format and distinct structures of informal science make it next to impossible for researchers to evaluate the quality of those experiences in the same way they can gauge formal schooling.

School assessments generally focus on cognitive measures, such as what a student knows and can demonstrate about particular content. In contrast, informal learning is dominated by noncognitive measures such as motivation, interest, and identity, according to Larry E. Suter, the National Science Foundation’s program director for informal science education. Moreover, traditional “gold standard” research methods such as randomized controlled trials can be detrimental to activities that base their strength on people’s choosing to participate, rather than being assigned.

Such research raises the risk, Mr. Suter said, that “if you touch it, you’re going to kill that thing you’re trying to study.”

That has led to some creative research alternatives. Alan J. Friedman, a former director and chief executive officer of the New York Hall of Science and the editor of the NSF’s 2008 framework for evaluating informal science education, recalls judging the effectiveness of an astronomy exhibit by the number of visitors who chose an astronomy poster over a different prize. Barbara N. Flagg, the director of the Multimedia Research consultant group in Bellport, N.Y., said she has used smudged museum walls indicating where visitors have touched exhibits, changes in Google and Amazon search terms over time, and phone interviews with parents and children.

“If you’re comparing this to other education research, you’ve got to turn back the clock 30 or 40 years,” said Kevin J. Crowley, the director of the University of Pittsburgh’s Center for Learning in Out-of-School Environments and an associate professor of education and psychology. “We are just now in the Wild West frontier, and people are just starting to gear up the longitudinal studies on how this will coalesce into a coherent narrative of how people learn science.”

Now, the tools being born of that creativity show the potential to link children’s education across a lifetime of different experiences, and in the process uncover more of how and what children learn than has ever been measured in a school test alone. Children, after all, spend more than 80 percent of their waking hours outside the classroom.

“The research has evolved,” Mr. Crowley said. “In the past, the great flaw of the informal-learning science was we looked at it in situations; we weren’t really looking at a lifelong trajectory in science learning. For the first time, we’re asking questions about how learning and participation are moving across place and across time.”

“Brenda’s” school performance, for instance, didn’t really show her scientific achievements or level of interest in the subject. Teachers reported the first-generation Haitian daughter of a single mother in Seattle had little interest in science and consistently “failed to engage” with the chemical-mixing tasks in her school lab.

Yet because researchers at the University of Washington, in Seattle, had been observing the 4th grader across formal and informal settings for more than 2,000 hours, they knew that school didn’t tell Brenda’s whole story. Not only did she regularly measure and mix chemicals and record the results for her perfume-making hobby, but she also had told the researchers she was considering becoming a chemist when she grew up.

“School science underrepresents her developing expertise,” Philip Bell, an associate professor of learning sciences at the university and the director of ethnographic and design-based research at the Everyday Science and Technology Group there, said during a recent lecture. “Just in terms of how people learn, our literatures don’t do justice to the varied pathways that people take through their experiences to make progress on things they care about.”

Vital Links

Learning Science in Informal Environments: People, Places, and Pursuits (2009)
The National Research Council of the National Academies, edited by Philip Bell, Bruce Lewenstein, Andrew W. Shouse, and Michael A. Feder
This landmark study by the National Academies’ Committee on Learning Science in Informal Environments documented evidence that children and adults do learn science outside of direct school instruction. Both designed science settings such as zoos or museums and spontaneous settings such as a walk in the park can help children understand science. The book laid the foundation for a more evidence-based approach to informal science education.

Surrounded by Science: Learning Science in Informal Environments (2010)
National Academies, Marilyn Fenichel and Heidi A. Schweingruber
A follow-up to the National Academies’ Learning Science in Informal Environments, this practitioner-focused guide provides case studies, framework tools, and other examples of how high-quality informal science education programs and exhibits can look.

Framework for Evaluating Impacts of Informal Science Education Projects (2008)
National Science Foundation, edited by Alan J. Friedman
Based on an NSF workshop on informal science education, this framework lays out the criteria for measuring informal science, based on a participant’s awareness, knowledge, or understanding of a science topic; engagement or interest in science; attitude toward science or careers in the field; changes in scientific behavior such as inquiry; and the improvement of specific skills related to science, such as experimenting or data analysis.

Measuring the Impact of a Science Center on Its Community (2011)
Journal of Research in Science Teaching, John H. Falk and Mark D. Needham
By studying Los Angeles residents and museum-goers before and a decade after a massive overhaul of the city’s California Science Center, researchers showed that the museum had increased the public’s understanding of and interest in the science covered in the new exhibits. The researchers used public understanding of homeostasis as a conceptual marker to track improved scientific understanding.

SOURCE: Education Week

Interest vs. Grades

That’s a dangerous disconnect, experts say, because mounting evidence shows that early engagement, even through informal pathways, eventually can lead to careers in the STEM fields of science, technology, engineering, and mathematics more surely than top grades in school.

In a 2006 study published in the journal Science, Robert H. Tai, an associate professor at the University of Virginia’s Curry School of Education, in Charlottesville, tracked thousands of students via the National Educational Longitudinal Study. He found that students who had only average grades in middle school but expressed interest in science were two to three times more likely to earn bachelor’s degrees in a science or engineering field 12 years later than high-achieving students who did not voice interest.

The landmark 2009 study that Mr. Bell co-wrote, “Learning Science in Informal Environments,” for the National Research Council, synthesized a vast array of research, including Mr. Tai’s, to conclude that “abundant evidence” exists that informal experiences can effectively support learning. What’s more, it found that those free-choice, low-stakes pathways can engage children from groups underrepresented in science fields: students in poverty, those from ethnic and language minorities, and women.

Still, while the number of research proposals submitted to the NSF about informal education has doubled in the past five years, from 300 to 600 annually, Mr. Suter said, the vast majority have been and continue to be from practitioners evaluating their individual programs, clubs, and exhibits, often using self-made assessments.

Better Alignment

In 2008, the Program in Education, Afterschool, and Resiliency at Harvard University and McLean Hospital in Boston found that of 64 widely used informal science evaluation tools, none met all five of the NSF’s five domains of informal learning: engagement and interest, attitude toward science and behavior, content knowledge, competence and reasoning, and career knowledge and acquisition.

Program founder Gil G. Noam and Rick Bonney, the director of program development and evaluation at the Cornell University Ornithology Lab, in Ithaca, N.Y., and the creator of several nationwide citizen-science projects, are working to change that. Mr. Noam has established the Assessment Tools in Informal Science database of existing tests. He said that he and Mr. Bonney are about a year away from using the tests to develop a single, unified assessment of informal science learning.

“Our big goal in all this is to bring into this field some proof that the informal science work makes a difference,” Mr. Noam said.

Likewise, Mr. Crowley’s organization, CAISE, has gathered five years’ worth of evaluation reports from NSF informal science projects—many never published—and has started to cull best practices and other lessons learned from across the many types of informal learning.

“Only recently has the learning theory caught up with those hunches, and now we’re at a place where we really can test design hypotheses based on what we know about the science of learning,” he said.

Ultimately, John H. Falk and Lynn D. Dierking, both professors in free-choice learning at Oregon State University, in Corvallis, hope the field will move toward more “synergy” studies, like Mr. Bell’s longitudinal work, that can explore how people learn science across the formal and informal experiences of their lives.

The Oregon State researchers now are monitoring how children in Portland, Ore., come to learn scientific concepts across the city’s many formal and informal science offerings during a four-year period. Portland provides a perfect microcosm for science learning, Mr. Falk said, because it has one science center, zoo, and public-broadcasting station and a few school districts that will allow the researchers to track individual students over time.

“In order to see real systemic improvement, you have to have a whole greater than the sum of its parts, and now the whole is less than the sum of its parts because there’s no alignment among these pieces,” Mr. Falk said. “We need to develop ways to better integrate practitioners across formal and informal education. We don’t understand how people navigate, make sense of, and make use of the resources in their community.”

This story was underwritten by a grant from the Noyce Foundation, at www.noycefdn.org.
A version of this article appeared in the April 06, 2011 edition of Education Week as Researchers Playing Catch-Up in Gauging Beyond-School Effects


Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and other jobs in K-12 education at the EdWeek Top School Jobs virtual career fair.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Reading & Literacy Webinar
Science of Reading: Emphasis on Language Comprehension
Dive into language comprehension through a breakdown of the Science of Reading with an interactive demonstration.
Content provided by Be GLAD
English-Language Learners Webinar English Learners and the Science of Reading: What Works in the Classroom
ELs & emergent bilinguals deserve the best reading instruction! The Reading League & NCEL join forces on best practices. Learn more in our webinar with both organizations.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Student Well-Being Opinion When Students Feel Unlucky, Teachers Can Help Change That Attitude
Mindsets matter when it comes to thinking about opportunity. Here’s what new research finds.
Paul A. O'Keefe
2 min read
Images shows a stylized artistic landscape with soothing colors.
Student Well-Being A Mental Health Screening Saved Students’ Lives in This District
A district that deployed a universal mental health screening was able to intervene immediately with five students who had suicide plans.
4 min read
Vector illustration of a counselor or psychologist holding a clipboard in one hand and an umbrella above in the other over an anxious woman who is tucking her head into her knees with a tangled line hovering above her head.
Student Well-Being Parents Worried About Their Kids' Mental Health See the Fix in New Schooling Options
Parents who say they are considering a change to their children's education identify mental health as a driving factor, a new report shows.
5 min read
Student walking down the stairs at her school.
iStock / Getty Images Plus
Student Well-Being Explainer More Students Are Getting Diabetes. Here's What That Means for the Classroom
More than a half million people under 20 could have the chronic health disorder by 2060, and they'll need support from schools.
8 min read
Conceptual image in blues: female student with diabetes wears glucose monitoring patch