This post is by Katherine Hayes, Chair of the Committee of External Research Review (CERR) and Coordinator for the School Experience Survey at the Los Angeles Unified School District (@LASchools).
Today’s post is the practitioner perspective on Monday’s post: Partnering on Survey Development: Benefits, Challenges, and Balancing the Trade-offs.
For the past nine years, Los Angeles Unified School District (L.A. Unified) students, parents, and staff have taken part in the School Experience Survey. This annual questionnaire is designed to collect information about the social and learning environments at their schools. The results are accessible through an online dashboard and are also used to inform district and state accountability documents. The survey is also an integral part of the School Report Card, which was created in 2009 by then-Superintendent Ramon C. Cortines to improve transparency and accountability and to give parents a greater voice in their child’s education.
Throughout the years and through various changes in leadership, there have been shifts in perspective and changes in the content and focus of the survey. Additional survey changes occurred in 2010, when L.A. Unified partnered with nine other urban districts in California to form CORE. Then starting in 2012, the Los Angeles Education Research Institute (LAERI) worked with L.A. Unified on the survey as well, providing additional modules and items. Many of these were administered beginning in 2015-16 as part of the LAERI/L.A. Unified joint college readiness work. These additional features include a high school counselor module; elementary and middle school counselor sections; and selected items for principals, teachers, and secondary students.
In developing and maintaining the survey, the District’s Office of Data and Accountability (ODA) considers all of the following:
Parents want simpler questions in both English and Spanish, and for surveys to be shorter. School-based staff have similar requests for student surveys.
Because principals use the survey to inform the Culture and Climate Goal in their annual School Plan for Student Achievement, they want the questions to be consistent.
Directors of ongoing projects also request consistency because they use the survey results to evaluate their programs.
Instructional and operational leaders ask for specific survey items to inform their work. As leaders change, the items they request may change.
While school staff want shorter surveys, administrator staff ask that items about their programs be included, resulting in longer surveys.
Principals want their data disaggregated by subgroup so they can gauge which groups feel connected to their schools.
Board of Education members use the survey results to project the long-term college and career success of L.A. Unified graduates.
Additionally, ODA needs to continually address and balance the following range of stakeholders’ perceptions and expectations about the survey:
District leaders expect schools to administer the surveys and use the data for accountability purposes.
Some local superintendents, who oversee a subset of schools, reward schools that obtain a response rate of 100 percent.
We shifted the survey administration window this year from January to November so the parents could complete the questionnaire while they were at schools for parent-teacher conferences.
Due to high expectations from leadership, school staff request real time reports of online survey response rates.
School staff request more time to administer the surveys.
Parents seem to prefer paper surveys which keeps the cost high. Last year, fewer than 20 percent of the parents completed the survey online. Our overall response rate for parents was 58 percent.
This year, LAERI worked closely with the district to develop the online surveys for staff and secondary students. In November, we administered the online surveys to staff, students and parents at 840 schools. Parents could opt to complete the survey on paper.
The context in which we develop and administer surveys makes our work with our research partner both rewarding and challenging. While LAERI seeks to develop and maintain items for research purposes, many district stakeholders are driven by the short-term needs and the pragmatic nature of the survey’s administration.
The School Experience Survey team consists of five analysts who conduct factor analyses and reliability tests on items, and who intend to pilot proposed changes to next year’s survey. However, the team is constrained by its size and the issues presented above. LAERI has always been available to provide technical counsel and training when requested.
The perseverance and patience of LAERI members as they strive to create and protect high-quality survey items, and the recognition by district staff that LAERI’s college-readiness work informs and enriches our own programs, fundraising, and planning (e.g. see blog by Carol Alexander or recent press coverage here and here) enables the continuation of the partnership. Our goals are the same. We strive to produce survey findings that can be used to make concrete improvements in practices and services provided to L.A. Unified students and families. The roads to that goal may not always converge, but our mutual respect and desire to improve the survey strengthens our resolve.