Widgetized Section

Go to Admin » Appearance » Widgets » and move Gabfire Widget: Social into that MastheadOverlay zone

Replication and Scaling Up: Using Process Evaluation to Understand Why Social Programs Do or Do Not Work

The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.

By Saunji Fyffe
November 26, 2018

More and more, public and non-profit leaders are conducting evaluations to identify the extent to which social programs are successful in achieving positive outcomes. However, social programs are seldom evaluated in ways that can determine whether they are operating or implemented as intended. A great deal of attention is being paid to outcome evaluation to assess whether a program is successful. Yet, conducting process evaluation helps to understand why programs succeed or fail. Looking at outcomes without analyzing how they were achieved presents a partial picture of a program and does not account for the organizational, programmatic and contextual characteristics that may be influencing outcomes or could be important for scaling and replicating efforts.

What is process evaluation and how does it differ from outcome evaluation?

Quality evaluations that follow a systematic approach usually will include both process and outcome evaluation. Process evaluation is distinct from outcome evaluation in that process evaluations occur during implementation and can be used to monitor and document operational factors. Through a process evaluation, quantitative (output) and qualitative data are collected on a regular basis to monitor and describe program activities. These data can help public and nonprofit leaders better understand how certain activities and elements of social programs influence program outcomes. For example, process evaluation can determine whether a program did or did not work due to implementation quality, changes that were made, barriers to delivery, challenges faced, resource investment, organizational characteristics, external factors and more.

In contrast, outcome evaluation occurs after implementation and measures a program’s results. Examining outcomes helps determine whether a program produced the changes or results in the target population that it was expected to achieve (e.g., changes in participant and/or target population’s knowledge, attitudes, skills, behaviors, or status). However, outcome evaluation data does not show what caused the results or provide information about why a program did or did not work.

Although process and outcome evaluation differ, they serve complementary purposes. When social programs achieve positive outcomes, it helps to understand which activities and characteristics contribute to success. Similarly, it is important to ask why certain programs fail to produce expected results. Without a rigorous and systematic process evaluation, it may be difficult to attribute outcomes to the program itself and not to contextual factors outside the program.

Why is process evaluation important?

Policymakers commonly assume a social program that is effective in one setting can produce the same results in other settings. However, social programs tend to be evaluated in a single setting, and so the results are not necessarily able to be generalized. For scaling and replicating social programs, it is important to understand the processes required to implement a program consistently and at a high level of quality. This is especially important because social programs often have different practitioners with varying levels of expertise implementing programs in different contexts.

How an intervention is implemented can affect outcomes. Hence, the success of scaling and replicating social programs often depends on implementation fidelity. Fidelity is a key indicator of implementation quality in community settings and refers to the extent an intervention corresponds to its original intention or is implemented as intended. Moreover, the integrity of program outcomes depends on knowing what aspects of the program were delivered and how well they were conducted. The lack of implementation fidelity can weaken outcomes, leading to flawed conclusions about program effectiveness, even causing potentially useful programs to appear unsuccessful.

Process evaluation advances understanding of the processes needed to maintain implementation fidelity. Examining implementation fidelity in process evaluation can help public and nonprofit leaders uncover variations in how social programs are adopted and adapted. Characteristics of participants and the context or situation in which programs are implemented also can be examined to help explain what may be influencing outcomes and inform plans to scale and replicate.

Conducting process evaluation in combination with outcome evaluation offers a more complete picture by linking program activities and observed outcomes with a higher degree of certainty. Determining whether a social program actually works or is achieving its expected outcomes is difficult without having evidence that a program has been implemented as intended.


Author: Saunji D. Fyffe, Ph.D., is a consultant specializing in program monitoring and evaluation. This article is based on her experience working with nonprofits to build capacity and implement performance measurement and management. For additional information, email [email protected]

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 5.00 out of 5)

Loading...

About

The American Society for Public Administration is the largest and most prominent professional association for public administration. It is dedicated to advancing the art, science, teaching and practice of public and non-profit administration.

Leave a Reply

Your email address will not be published. Required fields are marked *