Assessments: Do You Know What Are You Measuring?
The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.
By Ygnacio Flores, Tracy Rickman and Don Mason
October 22, 2018
Assessments are used by administrators around the world to measure how effectively and how efficiently an organization is performing. Goals and benchmarks are determined based on the measurements of performance. Public administrators, like those in public safety, have a large part of their image formed by the assessments conducted on their organizations. For instance, is the crime rate going up or down? How long does it take for an engine to respond to a fire or medical emergency? These assessments can affect personnel manning, procuring new equipment and also serve as a measurement of how well an agency is providing services.
Assessments are based on the analysis of data that is then transformed into information. There are a few issues with the process of developing intelligence from data. The first is that the data must be reliable — or the information suffers from the practice of garbage in, garbage out. The data itself should meet standards based on factual information that has been gathered. Is the data empirical, observed, experimental or is the data based upon reason or understanding —more of a realistic view? The second question posed is “does the analysis measure what is intended to be measured?” The third issue that should be reviewed is the use of data manipulation to shape inferences of the assessment. Many administrators tend to rely on the hubris of their administrative skills when reporting on their successes. Ensuring that the data is appropriate, reliable, factual and has not been manipulated to service an outcome that is desired rather than what is reflected in the analysis, is important.
An example would be of military commanders discussing the progress in a war. One commander asks, “Why is there little progress? We have dropped more bombs than the enemy. Thus, we should be winning against our enemy.”
The other commander responds with, “That may be so; however, how many bombs were on target?”
Therefore, many administrators are more concerned with how many bombs are being dropped rather than the bombs dropped on target. Adding to miss measurement are also those people who believe that they have dropped bombs in the first place. Assessments should focus on what questions are asked. What are we truly assessing. Are we assessing in our example, the progression of the war, or the effectiveness of the bombs dropped? Are we assessing the total number of bombs dropped or the damage caused by the bombs? We must ask the question, what are we measuring to begin the task of any assessment.
In law enforcement, there is little effect if extra patrols are added within a jurisdiction if the officers are assigned evenly across all patrol zones. The best effect on curbing crime is to assign officers to the neighborhoods with the most crime. This can be accomplished by analyzing the data on which crimes are being committed, at what times and at which locations. Private ambulance companies currently use this assessment model. They deploy their ambulances throughout a geographical jurisdiction at the times and locations where they anticipate the most services will be requested, based on past calls to service data. These ambulances may be parked in a local shopping center at 4:45 pm based on data that produced intelligence showing that there will be a higher call volume in the location within an hour before or an hour after 4:45 pm within a 15-mile radius of the shopping center.
One caution to heed in data analysis is to have clear definitions of what is being measured. For instance, consider the crime of larceny. Is the theft of a bag of chips being considered the same as the theft of a vehicle? Probably not. Nevertheless, few crime reports come with a glossary of terms or disclaimers of what is being considered in the data analysis. This is a reason why two different crime reports of the same city can have diametrically opposing findings. With this practice comes the shadow of who conducted or funded the research. It does not take much examination to discover if a report is biased to support an argument already in the public conversation. However, the measurement tools and questions to be asked and answered can be reviewed and further questions developed if required to meet the objectives of the survey. Thus, with assessments, there needs to be a continued focus on the questions, the measurements and how these are answered in the findings.
Assessments form a significant part of an administrator’s duties. Knowing where the rocks and shoals of measurement lie can aid in conducting the best scrutiny of the data being analyzed. Instead of preparing an assessment to say what you want it say, prepare an assessment that best answers the questions you are asking.
Authors: Ygnacio Flores, Tracy Rickman, and Don Mason are faculty members in the Public Safety Division at Rio Hondo College
(No Ratings Yet)
Loading...
Assessments: Do You Know What Are You Measuring?
The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.
By Ygnacio Flores, Tracy Rickman and Don Mason
October 22, 2018
Assessments are used by administrators around the world to measure how effectively and how efficiently an organization is performing. Goals and benchmarks are determined based on the measurements of performance. Public administrators, like those in public safety, have a large part of their image formed by the assessments conducted on their organizations. For instance, is the crime rate going up or down? How long does it take for an engine to respond to a fire or medical emergency? These assessments can affect personnel manning, procuring new equipment and also serve as a measurement of how well an agency is providing services.
Assessments are based on the analysis of data that is then transformed into information. There are a few issues with the process of developing intelligence from data. The first is that the data must be reliable — or the information suffers from the practice of garbage in, garbage out. The data itself should meet standards based on factual information that has been gathered. Is the data empirical, observed, experimental or is the data based upon reason or understanding —more of a realistic view? The second question posed is “does the analysis measure what is intended to be measured?” The third issue that should be reviewed is the use of data manipulation to shape inferences of the assessment. Many administrators tend to rely on the hubris of their administrative skills when reporting on their successes. Ensuring that the data is appropriate, reliable, factual and has not been manipulated to service an outcome that is desired rather than what is reflected in the analysis, is important.
An example would be of military commanders discussing the progress in a war. One commander asks, “Why is there little progress? We have dropped more bombs than the enemy. Thus, we should be winning against our enemy.”
The other commander responds with, “That may be so; however, how many bombs were on target?”
Therefore, many administrators are more concerned with how many bombs are being dropped rather than the bombs dropped on target. Adding to miss measurement are also those people who believe that they have dropped bombs in the first place. Assessments should focus on what questions are asked. What are we truly assessing. Are we assessing in our example, the progression of the war, or the effectiveness of the bombs dropped? Are we assessing the total number of bombs dropped or the damage caused by the bombs? We must ask the question, what are we measuring to begin the task of any assessment.
In law enforcement, there is little effect if extra patrols are added within a jurisdiction if the officers are assigned evenly across all patrol zones. The best effect on curbing crime is to assign officers to the neighborhoods with the most crime. This can be accomplished by analyzing the data on which crimes are being committed, at what times and at which locations. Private ambulance companies currently use this assessment model. They deploy their ambulances throughout a geographical jurisdiction at the times and locations where they anticipate the most services will be requested, based on past calls to service data. These ambulances may be parked in a local shopping center at 4:45 pm based on data that produced intelligence showing that there will be a higher call volume in the location within an hour before or an hour after 4:45 pm within a 15-mile radius of the shopping center.
One caution to heed in data analysis is to have clear definitions of what is being measured. For instance, consider the crime of larceny. Is the theft of a bag of chips being considered the same as the theft of a vehicle? Probably not. Nevertheless, few crime reports come with a glossary of terms or disclaimers of what is being considered in the data analysis. This is a reason why two different crime reports of the same city can have diametrically opposing findings. With this practice comes the shadow of who conducted or funded the research. It does not take much examination to discover if a report is biased to support an argument already in the public conversation. However, the measurement tools and questions to be asked and answered can be reviewed and further questions developed if required to meet the objectives of the survey. Thus, with assessments, there needs to be a continued focus on the questions, the measurements and how these are answered in the findings.
Assessments form a significant part of an administrator’s duties. Knowing where the rocks and shoals of measurement lie can aid in conducting the best scrutiny of the data being analyzed. Instead of preparing an assessment to say what you want it say, prepare an assessment that best answers the questions you are asking.
Authors: Ygnacio Flores, Tracy Rickman, and Don Mason are faculty members in the Public Safety Division at Rio Hondo College
Follow Us!