Measuring Performance Quantitatively

The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.

By Susan Paddock
June 9, 2017

The Nevada legislature recently debated teacher evaluation. Currently 80 percent of a teacher’s rating is based on observation; the remaining 20 percent is split between state testing data and locally-set student learning goals. Proposed legislation eliminates state testing data; performance would be evaluated almost wholly in qualitative terms.

By contrast, nationally many are urging quantitative teacher performance, based on how well students perform on standardized tests, and the removal of observational (qualitative) data from teachers’ evaluations. This affects school districts’ evaluations as well, since they are assessed in part on student test scores and graduation rates.

We see this drive for quantitative measurement in public administration as well, and the debate about the use of qualitative or quantitative data to evaluate individual and organizational performance.

There is strong evidence for using quantitative data. Accurately measuring outputs allows us to identify outcomes. Students who perform at grade level in reading are more likely to graduate, to find jobs that provide living wages and to seek post-secondary education. Professionals who visit and evaluate foster homes more often are more likely to identify clients who are neglected or abused. Police officers with more personal contacts in a neighborhood are more likely to identify potential perpetrators; neighborhood policing generally leads to lower crime rates.

Measuring outputs also highlights individual and organizational problems needing attention.

BUT (and this is important) – What if many employees fail to meet quantitative standards? What if an organization performs poorly? What if 50 percent of the teachers in a school district have classes where at least half of their students do not meet grade level or academic improvement standards? Does that mean the standards are wrong? If the standards are accurate (for example, measuring reading level), should we fire or discipline all the teachers who fail to meet standards? If the district has low graduation rates, is that due to poor teachers and administration, or to something else?

If qualitative data provide better information, and therefore we opt to rely on them, do we miss some important quantitative data? Having “satisfied” or “happy” trainees does not translate automatically into effective performance. A doctor’s patient relations may be unrelated to her ability to diagnose.

A secret to using quantitative data accurately is to measure against comparable individuals or organizations. If a teacher’s students are not meeting grade level standards, what is different between that classroom and others? Does that classroom have a higher teacher-pupil ratio? Less access to technology? More English language learners? A higher student turnover rate during a school year? These are some reasons why a teacher may have fewer students meeting standards.

On an organizational level, does the school district have more English language learners? More students who live in poverty? More students living in single parent homes or with parents who are not high school graduates? These all affect a district’s outcomes, and an appropriate evaluation compares a district against similar districts.

If a hospital has a higher death rate, does it have poor infection control procedures or instead does it receive patients who require more critical and intensive care? Are those patients ill from diseases that the hospital rarely treats? Hospitals which specialize in certain diseases or conditions experience more success in treating them.

If a city has a high crime rate, is that because of poor policing or because of its location (on a drug trade “route” for example), its poverty rate or its lack of adequate employment?

If a university has a lower rate of students graduating in 5 years, does it have more students who carry smaller credit loads for family or financial reasons? Or, does it fail to provide sufficient advising to students, has admitted students who are not prepared for postsecondary education or has inadequate resources?

I have been an opponent of hard-and-fast quantitative information in measuring organizational and individual performance, primarily because it is misused in at least four ways:

(1)    The wrong things are measured. As any car manufacturer knows, simply measuring the number of cars that are produced misses the effect of quality defects in organizational performance.

(2)    The wrong measures are used. Correlation is not cause-and-effect. A Likert scale may not be the best way to measure performance.

(3)    The right things are measured, but the population against which an individual or organization is compared is wrong. Evaluation of government programs for urban homelessness should compare cities with similar demographics, business profiles and even climate conditions.

(4)    The right things are measured in the right way, but the results are misused or ignored. A teacher whose students fail to meet grade-level standards may need more frequent observation, mentoring by a successful teacher or termination. Housing inspection errors may be due not to employees, but rather to limited staffing, poor training and supervision of staff, or outdated inspection standards. An organization must identify a response that addresses the cause.

Appropriate measures, appropriate comparable individuals or organizations, and appropriate consequences – these are the basics for effective use of quantitative performance evaluation. Quantitative measurement is important, but must be created, applied and used in a wise manner.

---

Author: Susan Paddock is a University of Wisconsin-Madison emeritus professor who lives and works in Las Vegas, NV. She is the former director of Certified Public Manager programs in Arizona and Wisconsin; has published in the areas of leadership, organizational development and human resources; and is an active student and researcher on what works in organizations. Email [email protected]

(1 votes, average: 5.00 out of 5)
Loading...