Meeting the Collective Impact Data Challenge
The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.
By Michael R. Ford
July 15, 2022
Last week my nonprofit students wrote thoughtful essays about the challenges of achieving collective impact in the nonprofit sector. A common theme was the difficulty of measuring macro-level progress, and building macro-level accountability systems for collective impact initiatives. For the uninitiated, collective impact refers to the use of multiple nonprofit organizations to collectively make progress on a societal problem.
On paper collective impact seems simple enough. There is a growing realization that government alone cannot address some societal needs, particularly when dealing with our most wicked problems, so the nonprofit sector is needed to fill in the gaps. More to the point, in a collective impact framework multiple nonprofits can address portions of a problem so that macro-level gains can be realized. The hope for collective impact is to avoid creating a population of nonprofits that are duplicating efforts and/or working at cross purposes.
Again, on paper it is easy to understand the logic. But, in practice getting a group of independent nonprofits with unique boards and missions to cooperate is challenging. Quite understandably, individual organizations are hesitant to give up power and resources to other organizations. This is where a strong umbrella organization, like a local United Way or community foundation, can play an organizing role. These organizations often set community goals and fund nonprofits working towards those goals. But, at least in my experience, there is rarely a sophisticated macro-level data framework tied to these initiatives. Nonprofits will release annual reports, collect and report internal data, share their financials and conduct surveys. But, these are tools for measuring organizational outputs, not community outcomes.
Measuring outcomes is harder. Sometimes we use census data, which is pretty broad, or rely on community survey data, which is often not fully representative of the populations nonprofit organizations are trying to serve. So what can we do better? I think it begins with utilizing K-12 education data. Changes in federal law over the past 20-plus years have spurred states and school districts to collect, track and release mountains of demographic and performance data for public school students. These data could be used for many more purposes than they are currently.
For example, in my city I have been pushing a collective impact framework that uses school district data to track community progress. Here is how it would work. First, an umbrella organization (in this case our local United Way) would need to determine all of their grantees’ programs that deal with school-aged children. Those nonprofits would have to agree to participate in the collective impact framework as a condition of funding. The umbrella organization would also need to create a common attendance form for their grantees to track student participation in every funded program. A common form would allow the umbrella organization to measure the total exposure to funded programming for every individual receiving services.
Parents of students using funded programming would need to consent to have their child’s individual school district data shared with a researcher tracking macro-level performance. The school district, or the state, would have to consent to release student-level data to an independent researcher working with the umbrella organization. With the framework in place, a researcher could annually:
- Measures each program’s individual impact on student performance indicators broken down by subgroup;
- Measures collective impact of all programs; and
- Provides actional feedback on what specific programs are working and why.
Such a framework would also free individual nonprofits from having to measure performance on their own, presumably freeing up more resources that can be applied directly to their mission.
The most common pushback I get from this idea is the problem of data confidentiality. While it is true that there are strict protocols for using student-level education data, all state education departments have a process that, when cleared through an Institutional Review Board, allows access to such data. No, it is not easy, but it is possible. The second most common pushback I receive involves fear. Organizations are fearful such data may not capture the progress they believe they are making. Umbrella organizations are fearful the data may reveal some long-funded programs are not making the impact they thought. My response to them is simple. Isn’t it better to know and course correct than to keep doing something that feels right but is not achieving your goals?
Author: Michael R. Ford is an associate professor of public administration at the University of Wisconsin Oshkosh, where he teaches graduate courses in budgeting and research methods. He frequently publishes on the topics of public and nonprofit board governance, accountability and school choice. He currently serves as the president of the Midwest Public Affairs Conference, and as an elected member of the Oshkosh, WI Common Council.
(2 votes, average: 5.00 out of 5)
Loading...
Meeting the Collective Impact Data Challenge
The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.
By Michael R. Ford
July 15, 2022
Last week my nonprofit students wrote thoughtful essays about the challenges of achieving collective impact in the nonprofit sector. A common theme was the difficulty of measuring macro-level progress, and building macro-level accountability systems for collective impact initiatives. For the uninitiated, collective impact refers to the use of multiple nonprofit organizations to collectively make progress on a societal problem.
On paper collective impact seems simple enough. There is a growing realization that government alone cannot address some societal needs, particularly when dealing with our most wicked problems, so the nonprofit sector is needed to fill in the gaps. More to the point, in a collective impact framework multiple nonprofits can address portions of a problem so that macro-level gains can be realized. The hope for collective impact is to avoid creating a population of nonprofits that are duplicating efforts and/or working at cross purposes.
Again, on paper it is easy to understand the logic. But, in practice getting a group of independent nonprofits with unique boards and missions to cooperate is challenging. Quite understandably, individual organizations are hesitant to give up power and resources to other organizations. This is where a strong umbrella organization, like a local United Way or community foundation, can play an organizing role. These organizations often set community goals and fund nonprofits working towards those goals. But, at least in my experience, there is rarely a sophisticated macro-level data framework tied to these initiatives. Nonprofits will release annual reports, collect and report internal data, share their financials and conduct surveys. But, these are tools for measuring organizational outputs, not community outcomes.
Measuring outcomes is harder. Sometimes we use census data, which is pretty broad, or rely on community survey data, which is often not fully representative of the populations nonprofit organizations are trying to serve. So what can we do better? I think it begins with utilizing K-12 education data. Changes in federal law over the past 20-plus years have spurred states and school districts to collect, track and release mountains of demographic and performance data for public school students. These data could be used for many more purposes than they are currently.
For example, in my city I have been pushing a collective impact framework that uses school district data to track community progress. Here is how it would work. First, an umbrella organization (in this case our local United Way) would need to determine all of their grantees’ programs that deal with school-aged children. Those nonprofits would have to agree to participate in the collective impact framework as a condition of funding. The umbrella organization would also need to create a common attendance form for their grantees to track student participation in every funded program. A common form would allow the umbrella organization to measure the total exposure to funded programming for every individual receiving services.
Parents of students using funded programming would need to consent to have their child’s individual school district data shared with a researcher tracking macro-level performance. The school district, or the state, would have to consent to release student-level data to an independent researcher working with the umbrella organization. With the framework in place, a researcher could annually:
Such a framework would also free individual nonprofits from having to measure performance on their own, presumably freeing up more resources that can be applied directly to their mission.
The most common pushback I get from this idea is the problem of data confidentiality. While it is true that there are strict protocols for using student-level education data, all state education departments have a process that, when cleared through an Institutional Review Board, allows access to such data. No, it is not easy, but it is possible. The second most common pushback I receive involves fear. Organizations are fearful such data may not capture the progress they believe they are making. Umbrella organizations are fearful the data may reveal some long-funded programs are not making the impact they thought. My response to them is simple. Isn’t it better to know and course correct than to keep doing something that feels right but is not achieving your goals?
Author: Michael R. Ford is an associate professor of public administration at the University of Wisconsin Oshkosh, where he teaches graduate courses in budgeting and research methods. He frequently publishes on the topics of public and nonprofit board governance, accountability and school choice. He currently serves as the president of the Midwest Public Affairs Conference, and as an elected member of the Oshkosh, WI Common Council.
Follow Us!