Performance or Effectiveness? A Critical Distinction for Teacher Evaluation

Guest post from Rod McCloy & Andrea Sinclair

Teacher evaluation has become a major focus of reform at the highest levels of education policymaking. The Obama administration awarded states more points for plans to improve teacher evaluation in their Race to the Top applications than for nearly any other policy area. The administration’s Blueprint for Reform for reauthorizing the Elementary and Secondary Education Act (i.e., NCLB) would require states to revamp teacher evaluation to receive significant amounts of federal funding. The administration has also allocated federal School Improvement Grant (SIG) money for “persistently low-performing” schools adopting the Transformation model, which requires an overhaul of current teacher evaluation practices. And now, most recently, to receive a waiver from the cornerstone requirement of NCLB—that all students be proficient in math and language arts by 2014—states must create new teacher evaluation guidelines. In all of these instances, it is required that the teacher evaluation system be revamped to include student achievement as a significant component.

Industrial-organizational (I-O) psychologists have much to offer this discussion. I-O psychologists apply psychological principles in an attempt to understand, predict, and improve workplace behavior. They touch on many topics, including personnel selection, training evaluation, job design, and organizational development. Arguably, though, the key concern for I-O psychologists is how to measure, predict, and improve job performance. We therefore are acutely aware of many problems associated with measuring individuals’ work performance. Even so, our field wandered about for decades using myriad measures as performance indicators, propagating substantial confusion in the research literature. In the early 1990s, however, a theory of job performance was proposed (Campbell, McCloy, Oppler, & Sager, 1993) that defines performance in a meaningful, explicit way and provides guidance to researchers and practitioners needing to choose performance measures for use in their studies and real-world settings, respectively.

The theory defines performance as

Performance on any job is complex; that is, it is not just “one thing” but instead consists of multiple, distinguishable components (it is multidimensional). One result of the complexity of job performance is that the notion of “overall job performance” is often not a meaningful concept. It is preferable to measure the components of performance separately. Decision-makers often invoke “overall job performance” because they need a single score upon which to base their decisions, but practical demands do not eradicate scientific reality.

The theory also makes a critical distinction between performance and three other concepts: effectiveness, productivity, and utility. Effectiveness is defined as

Here, we see the major problem with teacher performance being defined as a function of students’ standardized test scores. Rather than indexing teacher performance (behaviors under the teacher’s control), such a measure is an indicator of teacher effectiveness (the results of performing/not performing those behaviors).

To highlight the difference between performance and effectiveness, consider what it takes to be a high-performing sunglasses salesperson. The required characteristics are those that any good salesperson should possess: knowledge of the product line, an outgoing and friendly demeanor, and excellent interpersonal (e.g., instructing, social perceptiveness) and communication skills and abilities (e.g., active listening, oral expression). Possessing all of these characteristics, however, will NOT guarantee effectiveness, nor will their absence guarantee ineffectiveness! This is because effectiveness depends on factors extraneous to the person’s behavior, such as location (selling sunglasses in Binghamton/Seattle as opposed to Miami/Las Vegas). Imagine that we know the Seattle salesperson to be a much higher performer (better knowledge of the product line, more outgoing and friendly, etc.) than the Miami salesperson. Despite this, the Seattle salesperson yields lower sales than the Miami salesperson. Thus, the Seattle salesperson is a better performing—but less effective—salesperson.

What does this mean in terms of personnel decisions? Should we replace the Seattle salesperson in hopes of improving sales—perhaps by sending the Miami salesperson to Seattle? On the contrary, if anything, we should replace the low-performing Miami salesperson. Just imagine how effective a good performer would be in the desirable Miami sales market!

One should not infer from this example that we consider effectiveness to be unimportant. Indeed, it is the bottom line for most organizations. Nevertheless, it is at the level of performance that organizations have the greater capacity to influence results with appropriate interventions. The critical point is that we should not equate performance with effectiveness. This is not simply an exercise in semantics. They are separate, distinct concepts. Each is important, but they tend to be discussed interchangeably—to everyone’s detriment.

So what are the ramifications of the performance/effectiveness distinction for teacher evaluation? This will be the topic of the next blog post.

Dr. Rodney A. McCloy is a Principal Staff Scientist for the Human Resources Research Organization (HumRRO). With more than 20 years of experience conducting and directing personnel research, he serves as an in-house technical expert and a mentor to junior staff. His assessment and testing experience has spanned both cognitive and non-cognitive domains and has involved several large-scale assessment programs (Armed Services Vocational Aptitude Battery, National Assessment of Educational Progress, General Aptitude Test Battery). He has served as adjunct faculty at both The George Washington University and George Mason University. He is a Fellow of the American Psychological Association (APA) and the Society for Industrial and Organizational Psychology (SIOP). He received his Ph.D. in Industrial-Organizational Psychology from the University of Minnesota in 1990.

Dr. Andrea L. Sinclair is a Senior Scientist in HumRRO‘s Validity Investigations for Education and the Workplace (VIEW) Program. She conducts research in education, government, military, and private sector settings with a particular focus on performance measurement and program evaluation. She regularly develops performance measurement instruments, surveys, and observation and interview protocols for use in schools. In addition, she regularly advises clients on the validity and reliability of their assessment systems and on the development of competency models. She received her Ph.D. in Industrial-Organizational Psychology from Virginia Tech in 2003.

Reference:
Campbell, J.P., McCloy, R.A., Oppler, S.H., & Sager, C.E. (1993). A theory of performance. In N. Schmitt & W. Borman (Eds.), Personnel selection in organizations (pp. 35-70). San Francisco, CA: Jossey-Bass.

“Job Performance” Measures from the Research Literature (from Campbell, McCloy, Oppler, & Sager, 1993, p. 36):

• Time to complete a training course
• Grades or achievement test scores earned in training
• Number of errors made in a simulator
• Number of Tinkertoy figures assembled in a 45-minute experimental session
• Number of one-minute marketing interviews completed outside a shopping center in one day
• Number of pieces produced
• Number of defective pieces produced
• The total or average cost of the pieces produced
• Number of proposals written
• Total value of contracts won
• Total value of sales
• Number of grievances or complaints incurred
• Length of tenure in the organization
• Total days absent
• Salary level
• Promotion rate within an organization
• Percentage over budget
• Supervisor, peer, subordinate, or self ratings of “overall” performance
• Scores on a paper-and-pencil job knowledge test
• Scores on a professional certification test
• Number of citations in the citation index over a 3-year period
• Promotion rate within an organization
• Number of refereed journal articles published in a 6-year period