Making Use of User Satisfaction Surveys
The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.
By Nathan Favero
January 11, 2020
More and more public organizations are conducting user satisfaction surveys. Online tools like SurveyMonkey make it easier than ever to create a simple survey, but just because it is easy to send out a survey doesn’t mean it is easy to make smart use of satisfaction surveys.
Here are five tips for getting the most out of satisfaction surveys:
1. Think carefully about your sample
People tend to focus too much on sample size and too little on who is in the sample. A large sample does you little good if only the most unhappy users bothered to fill out the survey, causing the results to give a skewed view of what users think.
So how can you get a good sample?
First, think about who you should survey. Maybe you want to learn more about people who dropped out of your program prematurely. Or maybe you want a broad cross-section of program participants. Decide what you are hoping to learn from your survey. Then, based on these goals, choose who you will target with your survey.
Second, think about how you will invite people to participate, with the goal of maximizing the response rate among those you invite. Sometimes, calling 80 randomly-chosen people and interviewing them over the phone is better than sending an email invitation to every program participant. Just think about how flooded your email inbox is with invitations to participate in various online surveys. What will make your survey stand out? You want to avoid getting responses from only the users with unusually strong opinions or too much free time on their hands.
2. Know what users know… and don’t know
Depending on the type of organization you work in, your typical user may not have strong or well-informed opinions about your services. That doesn’t mean they won’t have valuable information to offer, but it does mean you need to take extra care in designing your survey and interpreting your results.
Construct your survey in a way that emphasizes topics you think your respondents are knowledgeable about, such as their own experiences. Give them the option to say, “I don’t know,” or, “Not applicable,” if you are asking about something that will not be relevant to all respondents. And be sure to offer respondents a place to share their opinions about whatever is most important to them.
3. Test your survey by piloting it
It’s not easy to write good survey items. Most respondents won’t read the survey as carefully as you, so your questions and response options have to be clear and concise. Avoid fancy words and technical jargon. Don’t push people toward one view or the other, even if you have your own opinion about whatever you’re asking. And keep the survey as short as possible.
The best way to make sure you have written a good survey is to pilot it. Send a draft of the survey to your coworkers and maybe a few trusted users. Keep track of how long it takes them to complete the survey. Then ask about what was unclear, what was missing, and what could otherwise be improved.
4. Be mindful of varying expectations and experiences
Remember that no two users are exactly alike. If one user is happier than another, the first user may have received better service, or maybe they just had lower expectations. One of the benefits of user surveys is that they can help you understand what it is that different users expect from you.
It is often helpful to gather some basic demographic characteristics about your survey respondents so that you can see if different subgroups have different experiences or expectations (assuming you have a reasonably-sized sample for each subgroup)
5. Think about anonymity and incentives
When you invite people to participate in the survey, clearly tell them whether or not the survey is anonymous. Be careful about promising anonymity if you have a small number of users or ask demographic questions that will make it possible to narrow down who a specific respondent is. If anonymity is really important, you might want to hire an outside firm to conduct the survey, since they can assure respondents that their identities will be kept secret from your organization.
If the survey isn’t anonymous, come up with clear rules about who will get to see the data, and explain these rules to survey participants. Then, of course, make sure you follow these rules.
Finally, user satisfaction data can serve as one measure of performance, but be careful about creating incentives for employees to get good marks on the survey. Incentives can drive people to higher performance, but there can also be unintended consequences when incentives are tied to particular metrics. For example, personnel may encourage users to answer the survey favorably, or employees might change behavior in ways that improve user satisfaction but that harm other aspects of organizational performance.
The bottom line here is to spend some time thinking about how various people in the organization might perceive the survey so that you get honest responses and encourage positive behavior among personnel.
Author: Nathan Favero (nathanfavero.com) is an assistant professor in the School of Public Affairs at American University. His research focuses on public management, education policy, social equity, and research methods. Email: [email protected]. Twitter: @favero_nate
(No Ratings Yet)
Loading...
Making Use of User Satisfaction Surveys
The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.
By Nathan Favero
January 11, 2020
More and more public organizations are conducting user satisfaction surveys. Online tools like SurveyMonkey make it easier than ever to create a simple survey, but just because it is easy to send out a survey doesn’t mean it is easy to make smart use of satisfaction surveys.
Here are five tips for getting the most out of satisfaction surveys:
1. Think carefully about your sample
People tend to focus too much on sample size and too little on who is in the sample. A large sample does you little good if only the most unhappy users bothered to fill out the survey, causing the results to give a skewed view of what users think.
So how can you get a good sample?
First, think about who you should survey. Maybe you want to learn more about people who dropped out of your program prematurely. Or maybe you want a broad cross-section of program participants. Decide what you are hoping to learn from your survey. Then, based on these goals, choose who you will target with your survey.
Second, think about how you will invite people to participate, with the goal of maximizing the response rate among those you invite. Sometimes, calling 80 randomly-chosen people and interviewing them over the phone is better than sending an email invitation to every program participant. Just think about how flooded your email inbox is with invitations to participate in various online surveys. What will make your survey stand out? You want to avoid getting responses from only the users with unusually strong opinions or too much free time on their hands.
2. Know what users know… and don’t know
Depending on the type of organization you work in, your typical user may not have strong or well-informed opinions about your services. That doesn’t mean they won’t have valuable information to offer, but it does mean you need to take extra care in designing your survey and interpreting your results.
Construct your survey in a way that emphasizes topics you think your respondents are knowledgeable about, such as their own experiences. Give them the option to say, “I don’t know,” or, “Not applicable,” if you are asking about something that will not be relevant to all respondents. And be sure to offer respondents a place to share their opinions about whatever is most important to them.
3. Test your survey by piloting it
It’s not easy to write good survey items. Most respondents won’t read the survey as carefully as you, so your questions and response options have to be clear and concise. Avoid fancy words and technical jargon. Don’t push people toward one view or the other, even if you have your own opinion about whatever you’re asking. And keep the survey as short as possible.
The best way to make sure you have written a good survey is to pilot it. Send a draft of the survey to your coworkers and maybe a few trusted users. Keep track of how long it takes them to complete the survey. Then ask about what was unclear, what was missing, and what could otherwise be improved.
4. Be mindful of varying expectations and experiences
Remember that no two users are exactly alike. If one user is happier than another, the first user may have received better service, or maybe they just had lower expectations. One of the benefits of user surveys is that they can help you understand what it is that different users expect from you.
It is often helpful to gather some basic demographic characteristics about your survey respondents so that you can see if different subgroups have different experiences or expectations (assuming you have a reasonably-sized sample for each subgroup)
5. Think about anonymity and incentives
When you invite people to participate in the survey, clearly tell them whether or not the survey is anonymous. Be careful about promising anonymity if you have a small number of users or ask demographic questions that will make it possible to narrow down who a specific respondent is. If anonymity is really important, you might want to hire an outside firm to conduct the survey, since they can assure respondents that their identities will be kept secret from your organization.
If the survey isn’t anonymous, come up with clear rules about who will get to see the data, and explain these rules to survey participants. Then, of course, make sure you follow these rules.
Finally, user satisfaction data can serve as one measure of performance, but be careful about creating incentives for employees to get good marks on the survey. Incentives can drive people to higher performance, but there can also be unintended consequences when incentives are tied to particular metrics. For example, personnel may encourage users to answer the survey favorably, or employees might change behavior in ways that improve user satisfaction but that harm other aspects of organizational performance.
The bottom line here is to spend some time thinking about how various people in the organization might perceive the survey so that you get honest responses and encourage positive behavior among personnel.
Author: Nathan Favero (nathanfavero.com) is an assistant professor in the School of Public Affairs at American University. His research focuses on public management, education policy, social equity, and research methods. Email: [email protected]. Twitter: @favero_nate
Follow Us!