The use of analytic tools to predict student performance is exploding in higher education, and experts say the tools show even more promise for K-12 schools, in everything from teacher placement to dropout prevention.
Use of such statistical techniques is hindered in precollegiate schools, however, by a lack of researchers trained to help districts make sense of the data, according to education watchers.
Predictive analytics includes an array of statistical methods, such as data mining and modeling, used to identify the factors that predict the likelihood of a specific result. They’ve long been a standard in the business world—both credit scores and car-insurance premiums are calculated with predictive analytic tools. Yet they have been slower to take hold in education.
“School districts are great at looking annually at things, doing summative assessments and looking back, but very few are looking forward,” said Bill Erlendson, the assistant superintendent for the 32,000-student San José Unified School District in California. “Considering our economy survives on predictive analytics, it’s amazing to me that predictive analytics don’t drive public education. Maybe in education it’s considered a luxury, but it shouldn’t be; it should be a foundation for making decisions.”
Experts in predictive analytics in higher education and business say education may have a long way to go to develop the data infrastructure and staff capacity to make the tools useful on a broad scale.
“Good quantitative researchers are as hard to find in academia as Farsi linguists are for the military; we do not train enough researchers to work with these methods,” said Phil Ice, the vice president of research and development for the 90,000-student online American Public University System, in Manassas, Va. “There are plenty of numbers people, but they work in the corporate sector, and they don’t know how to apply it to the education sector. You have to understand the pedagogy, the social issues around education, and you have to understand the numbers.”
Predictive models might show the likelihood that a student with certain characteristics will excel in college or a teacher’s credentials and instructional style will gel in a new school, but they aren’t guarantees of specific results for individual teachers or children. Administrators have to walk a thin line between targeting support and changing expectations and opportunities based on predicted risks, according to Yeow M. Thum, a senior research fellow at the Northwest Evaluation Association, in Portland, Ore., who studies education growth modeling.
New Territory
“Risk management and topics like that are really foreign to education research and management,” Mr. Thum said. “How people couch the accountability question will affect the results. The problem itself is so fluid—and we should be glad it’s fluid when it comes to human capacities—but it is so malleable, [prediction] is going to be a very, very difficult thing to do.”
One district that is taking the plunge into using predictive analytics for policymaking is San José. The district is modeling high school graduation and college-going trends based on 15years’ worth of student academic, behavioral, social development, and health data, as well as information on school climate from teachers, parents, and students. The district is finalizing a risk-assessment protocol that identifies changing issues that contribute to a student’s risk of dropping out at different grades.
Each student has an individual profile, including the status of key indicators like benchmarking and summative test scores, behavior, attendance, and health. Student data will be updated daily and monitored for accuracy, Mr. Erlendson said, but is not used for student report cards.
“It’s not quite a credit score, but it could be” eventually, as more indicators are added to the mix, he said. “Right now, we’re on the cutting edge of what a student dashboard looks like. The indicators will be very clear: test scores, behavior, academics, health. The data there are going to be pretty solid. If the student is failing or is not showing up, that will be pretty obvious.”
Similarly, in 2007, Hamilton County schools, which serve 41,000 students in and around Chattanooga, Tenn., used districtwide data on student demographics, test scores, attendance, and other information, comparing the graduation and dropout rates of students with different characteristics at each grade level to develop an “on track” predictor of each student’s likelihood to graduate. Kirk Kelly, the director of accountability and testing for Hamilton County schools, said individual schools have been focusing on the risk factors that carried the most weight for them.
For example, some elementary schools found that students who failed early grades such as kindergarten or 1st grade were likelier to drop out in high school. Thus, elementary schools began more-intensive monitoring and remediation for students at risk of failing an early grade, and the district started “adult” high school programs for students considered overage for their grades.
It’s been four years since the district started using the tool to target interventions, Mr. Kelly said, and, “we’ve seen the tool starting to bear fruit.” The four-year graduation rate has increased steadily from 70.9 percent in 2008, the first school year of implementation, to 80.2 today, and the rate of students dropping out in each school year tumbled from 6.4 percent in 2008 to 1.8 percent today.
San José, along with the Dallas, Pittsburgh, and Philadelphia districts and the New York-based New Visions for Public Schools charter school network, are taking part in a three-year project analyzing school feeder systems to identify the elements at each school level that predict a student’s understanding of college entrance, readiness for college content, and ability to complete a degree.
Data System Demands
Both Mr. Erlendson and Mr. Kelly cautioned that districts must have a well-developed data system with several years’ of data in multiple areas to get a nuanced prediction.
“We had to look at many things that could be predictive at different times,” Mr. Erlendson said. “Public education is such a fluid thing, it’s like trying to lasso an amoeba.”
That complexity causes problems when trying to connect a child’s experiences in elementary school with success in college 15 years later, Mr. Thum said. “We know that predictions on the near term are very useful; predictions in the far term are fraught with problems,” Mr. Thum said.
Moreover, many predictive-analytics systems used in business, like that which provides a credit score, use the data to rank people based on the likelihood of a specific behavior, something Mr. Thum said generally is not appropriate for education.
“Very often, the folks hired into key policymaking positions probably do not have as much patience as they ought to have,” Mr. Thum said. “Often ranking is all they are after; they’re not looking at measuring student learning.”
Yet sometimes, ranking can be useful, such as when a principal is trying to pick the right new teacher to fit into a school staff.
Teresa Khirallah, the senior director of Peak Preparatory, a K-12 charter school in eastern Dallas, has seen the hiring process work with and without predictive tools. When she took the administrative reins of the school four years ago, principals found new teachers through personal applications from teachers or the central candidate pool for the school’s parent organization, Uplift Education of Dallas.
“We spent a lot of time weeding people out, not really knowing from a five-minute phone conversation or email if they were the right person to continue this process,” Ms. Khirallah said. “There was just a lot of time spent on individuals who by the end of the hiring process you realized did not match the mission or would not be a good fit with your kids.”
Two years ago, with the 17-school network growing 25 percent or more a year, Uplift’s Chief Executive Officer Yasmin Bhatia overhauled the network’s hiring process in advance of hiring 160 teachers. Uplift, working with the Emeryville, Calif.-based 3D Group, analyzed 44 different tasks that Uplift teachers perform, using a combination of surveys, interviews, and classroom observations with the teachers considered to be exemplary based on qualifications, experience, and recommendations.
The teachers rated each task on its importance to their daily job, the equipment or materials needed, and whether a new teacher should enter able to do the task or expect to learn on the job. Teachers also related examples of their biggest successes and mistakes in performing different teaching tasks.
After crunching the data, Uplift had a list of 29 ranked tasks in five teaching areas which formed the backbone of a new teacher-hiring process. A potential candidate’s responses to essay questions can be rated according to those indicators, and principals use formal interview questions, with answers also rated to make it easier to compare candidates.
“These are very specific situations we are asking candidates to describe for us; you cannot fluff our questions,” Ms. Bhatia said. “If you see that this person is only a 2.5 out of a 5 [in one of the five areas], it forces the discussion of, ‘Am I going to make the trade-off or am I going to hold off and keep looking for someone who is a better fit?’”
For principals, the change has clarified the hiring process, Ms.
Khirallah said. “What I’m able to do as a principal is to spend more time talking with and training the right people, rather than spending a lot of time weeding people out.”
While Uplift is still measuring the achievement effects of teachers hired under the new system, Ms. Khirallah said she has already seen a decline in teacher turnover.
Looking Ahead
Ms. Bhatia said the network has started to use the initial task scores to tailor professional development based on a teacher’s strengths and weaknesses, and the schools hope to eventually have more fine-grained data to help match teachers to specific subjects or grade levels. The May issue of the Harvard Education Letter noted Uplift’s predictive analytics show promise in helping districts match educators to the schools where they will be most effective.
Higher education has started to explore the validity of using predictive analytics for long-term goals like college success. For example, the Western Interstate Commission for Higher Education has won a $1 million grant to develop a Predictive Analytics Reporting Framework using data from more than 400,000 students in six college systems with online students. The colleges are looking at 34 factors that contribute to a student’s ability to enter and succeed in college.
“I think that the K-12 environment may even be a richer source of material than the university setting,” said Mr. Ice, the principal investigator. “We know so much more about what our students are doing and where they are coming from,” he said.
