Metrics that Count: Supporting Evidence-based Approaches
The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.
By Thomas Poulin
December 22, 2017
For decades, public administration has pursued evidence-based approaches to decisionmaking, seeking to optimize such processes in public agencies. This will only work if the evidence is based on some form of objective metrics. We must feel comfortable the metrics are valid (measuring what we wish to measure) and if they are reliable (arriving at generally identical “readings” each time). This is not always practiced effectively in the professional environment.
- Long ago, in a training session, a senior executive told us we were one of the top ten departments in our discipline in the nation. When I asked how that was determined, he said this was his belief — it was up to me to prove him wrong. (I yielded.)
- Working with professionals from organizations across the country, I have routinely heard of people asked to conduct “research” to support a decision that was already made. This type of process is more about developing a marketing-based approach to “sell” an idea than it is to make an evidence-based decision on what approach is most suitable for the situation.
- Years ago, a local law enforcement agency installed cameras and facial recognition software using a combination of grant money and local funds. When asked by elected officials and senior city leadership how law enforcement would measure the effectiveness, the Deputy Chief of Police stated if he they caught someone, it showed the system worked. If they didn’t catch anyone, it showed the system was so effective that criminals elected to not come to the area, so the project would be a success. Two paths to success, with no potential for failure.
Clearly, these are examples of where objective metrics were not used, which might have contributed to problems at some stage. In the end, such approaches might succeed, but that might be attributable as much to good fortune than to anything else. Most professionals and academics agree the use of evidence-based approaches using objective metrics should maximize the potential for success.
There is no single metric that can be use in all settings. One of the challenges faced in both the professional realm and academia is finding objective metrics that meet your needs. Only then can you consider how to collect and analyze the data. There are differing approaches which might be used.
- Hard Data: It is relatively easy to determine costs or workhours, as we can identify and collect hard data in the form of specific numbers. An organization is often already collecting such data. Sometimes it might not be, and it is all too common for researchers to have to develop a data collection system to collect sufficient data to know where they are in the present. At other times, they might find they have been collecting data for years, but not using it for analysis and evaluation of performance. Academics often use this type of secondary data set for their work – perhaps organizations should consider finding a use for this data in their own decisionmaking processes.
- Construct: How do you measure something that is not
concrete? For example, how do you measure job satisfaction or motivation? Simply asking for an opinion might not yield honest results. Consequently, we can construct a metric. In this case, we might look at taking the numbers of grievances, the number of sick days, and the turnover rate for the organization, using that hard data to arrive at a single metric of job satisfaction. This is a common approach in research when one is looking at something intangible. It can be a complex process to create and validate, but it can contribute to effective, objective metrics to support decisionmaking.
- Proxy: A proxy is the use of one metric to substitute for another. In one instance, I was tasked with comparing response times in a fire department, but there were many concerns with the accuracy of the available data. Instead of using times, I used mileage. It was easily available, and it provided a proxy for response (i.e., longer response distance would equate to lengthier response times). However, approach proxies carefully. Often, research in human resource management might seek to measure job satisfaction by using a proxy such as asking if the individual intends to leave the organization within the next X years. The presumption is if the employee is satisfied, they will not be intending to leave. However, in many public sector organizations, the pension system is crafted in such a manner that employees are “trapped” by the system – they might wish to leave, finding an organization where they would feel more fulfilled, but cannot because of the potential loss of status, compensation, and benefits. This does not argue against the use of such a proxy, but does suggest that proxies must be approached warily.
Clearly, this is only a simplified overview of the topic, but it should drive home a few specific points. First, organizations may, even if only unintentionally, be making decisions on flawed metrics, which can contribute to sub-optimal results. As evidence-based decisionmaking requires objective metrics, this should be a concern for us all. Second, we can often find or develop metrics that will meet our needs. At times, the data is already there, we are simply not using it. In other instances, we will have to develop the metrics. While that can be challenging at first, like any skill set, it comes with time. By doing so, we create an objective means to assess where we were, where we are and where we want to be. Developing a means to look at our work in an objective, detached manner can help to make us more formidable as researchers and more credible as professionals in public administration, contributing to our success in multiple arenas.
Author: Thomas E. Poulin, PhD, MS(HRM), MS(I/O Psych.), EFO serves on the core faculty of Capella University’s public administration programs. Prior to that, he served in local government for over three decades. He may be contacted at [email protected].
(1 votes, average: 5.00 out of 5)
Loading...
Metrics that Count: Supporting Evidence-based Approaches
The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.
By Thomas Poulin
December 22, 2017
For decades, public administration has pursued evidence-based approaches to decisionmaking, seeking to optimize such processes in public agencies. This will only work if the evidence is based on some form of objective metrics. We must feel comfortable the metrics are valid (measuring what we wish to measure) and if they are reliable (arriving at generally identical “readings” each time). This is not always practiced effectively in the professional environment.
Clearly, these are examples of where objective metrics were not used, which might have contributed to problems at some stage. In the end, such approaches might succeed, but that might be attributable as much to good fortune than to anything else. Most professionals and academics agree the use of evidence-based approaches using objective metrics should maximize the potential for success.
There is no single metric that can be use in all settings. One of the challenges faced in both the professional realm and academia is finding objective metrics that meet your needs. Only then can you consider how to collect and analyze the data. There are differing approaches which might be used.
Clearly, this is only a simplified overview of the topic, but it should drive home a few specific points. First, organizations may, even if only unintentionally, be making decisions on flawed metrics, which can contribute to sub-optimal results. As evidence-based decisionmaking requires objective metrics, this should be a concern for us all. Second, we can often find or develop metrics that will meet our needs. At times, the data is already there, we are simply not using it. In other instances, we will have to develop the metrics. While that can be challenging at first, like any skill set, it comes with time. By doing so, we create an objective means to assess where we were, where we are and where we want to be. Developing a means to look at our work in an objective, detached manner can help to make us more formidable as researchers and more credible as professionals in public administration, contributing to our success in multiple arenas.
Author: Thomas E. Poulin, PhD, MS(HRM), MS(I/O Psych.), EFO serves on the core faculty of Capella University’s public administration programs. Prior to that, he served in local government for over three decades. He may be contacted at [email protected].
Follow Us!