Widgetized Section

Go to Admin » Appearance » Widgets » and move Gabfire Widget: Social into that MastheadOverlay zone

An Emerging Innovation in Policy Evaluation: From Theory to Practice

By Steven E. Wallis

Within the broad spectrum of social sciences, the world of policy is unique. There seems to be a refreshing candor here that is not found in other fields. Perhaps this is because policy people accept that the policy process involves difficult research (where the data is sometimes simply unavailable), analysis (constrained by the limits of the human mind), creation of a formal policy (torn by political wrangling), and implementation (under impossible circumstances and incomplete funding). We are using primitive tools to try and solve the complex problems of a modern world. Yet, we push forward because we know that our work is important.

Many new tools have been gaining ground. Systems thinking, for example (along with its close cousin complexity theory), has been proposed as a way to gain new insights into policy situations. However, that field is full of esoteric notions such as emergence, bifurcation, cybernetics, and fitness landscapes. Even the experts admit that their processes are not easy to understand or use. Most of us have trouble enough just trying to make sense of the world we live in; we need a tool that is easier, not one that is more complex.

wallis 4 Fortunately, over the past 60 years, an interesting stream of research has been finding new insights into the way we think, communicate, create theories, and make effective policies. The main idea is that there is an underlying structure to the way we understand the world. Individuals who are capable of complex patterns of thinking tend to be more collaborative and more successful as leaders. They are also able to adapt more effectively to complex and dynamic situations. Importantly, we can measure the complexity of that structure. For example, integrative complexity has been used to analyze writing samples and quantify the complexity of those samples on a seven-point scale. Some of that work falls under the heading of political psychology and links may be found at: http://en.wikipedia.org/wiki/Integrative_complexity.

A variation of that work has been applied to analyze academic theories. Research shows that theories of measurably higher systemic structure are more likely to be effective in practical application. This newer method, called Integrative Propositional Analysis (IPA), has proven useful for analyzing a range of conceptual systems including theories, ethics, and policy models. Here, by policy model, I am referring to the “map” we use to describe and make sense of a policy situation.

Briefly, IPA is a simple six-step process for deconstructing policies into their component propositions, mapping those propositions graphically, quantifying the complexity (number of concepts), and quantifying the causal relationships between those concepts to determine the systemicity (extent that the concepts exist in systemic relationship to one another). IPA is a complexity-based approach that is relatively simple to use.

My book, Avoiding Policy Failure: A Workable Approach, presents comparative case studies where policy success is contrasted with policy failure. The results show how policy models that are more complex and more systemic have historically been more successful in application. Those results, and the history of the emerging sub-field, suggest that we can use IPA to predict the success of policy models. Indeed, we now have a tool for making better policy models (at least from a structural perspective).

In retrospect, the perspective made perfect sense because we live in a world that is highly complex and highly systemic. For example, we are made of biological systems, we use communication systems, and we are imbedded in a variety of social systems. In short, in a world made of systems it makes sense to understand that world using policy models that are also systemic. According to the research, IPA should provide a useful tool for analyzing policy models from a systems perspective. It can be used to identify strengths and weaknesses of understanding, predict success, support collaboration, focus conversations, and clarify directions for research. While it will not identify “unknown unknowns,” it will point the direction in which they may be found.

Importantly, because each policy model is quantified in terms of its complexity and systemicity, groups that are using IPA to develop more effective models will have an objective indicator of progress. Meaning, we can look at the model at the start of the day, compare it with the model at the end of the day and know what we have added value to the policy development process. We have not been simply arguing around in circles.

wallis 3

In a world hungry for new ways of dealing with complex issues, there is a growing interest in practical uses for this emerging methodology. I have engaged in conversations with many colleagues. I teach IPA through the Foundation for the Advancement of Social Theory (FAST), where FAST fellows use IPA for research. Naturally, I have presented on IPA and facilitated workshops at national and international conferences. Also, IPA is central to my posting on the Fulbright Specialist Roster. Currently, I am developing projects with a number of universities where I anticipate using IPA to facilitate policy conversations to build better policy models. In short, IPA is moving rapidly from theory to practice.

I authored a recent article in the Emergence: Complexity & Organization An International Transdisciplinary Journal of Complex Social Systems titled “How to Choose Between Policy Proposals: A Simple Tool Based on Systems Thinking and Complexity Theory,” which explains how IPA can be used as a simple tool to empower those who lack a voice in the policy development process.  Over the coming years, I look forward to collaborating with a variety of scholars and practitioners around the world to move IPA from theory to practice. In concert with existing methods (e.g. confirming the quality of the data), IPA will provide a new perspective and open the door for illuminating conversations and insights. I hope IPA will prove useful for you.


Steven E. Wallis, Ph.D., Director of FAST, mentors doctoral candidates through the mental chaos of their dissertations at Capella University, and is on the Fulbright Specialist Roster. Steve has about a decade of experience as an Organization Development consultant, coach and trainer in Northern California where he is a (mildly) competitive Epee fencer and enjoys riding through the wine country on his Ural side-car rig with his wife Cecily. He can be reached at: [email protected]

1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 5.00 out of 5)

Loading ... Loading ...

6 Responses to An Emerging Innovation in Policy Evaluation: From Theory to Practice

  1. Jim Butt Reply

    December 23, 2013 at 3:40 pm

    The recent changes to the patimes blog pages wherein images cycle are causing pages to jump up and down with the changing image sizes. The html around the images should cause them to scale to a fixed size. Good luck.

    • PATIMES Reply

      December 24, 2013 at 9:58 am

      Thanks, Jim. We’ve noted this problem to our support team. For now, we are limiting each article to one image.

  2. Pingback: An Emerging Innovation in Policy Evaluation: From Theory to Practice | PA TIMES Online | JimButt.Info

  3. Steve Wallis Reply

    November 12, 2013 at 7:57 am

    For more related approaches, check out our new site: http://justaskmatt.wordpress.com/

  4. Steve Wallis Reply

    October 21, 2013 at 4:59 pm


    That someone would probably be me.

    Briefly, each concept contained in the policy model counts toward the complexity of the model. A policy containing ten concepts would have a complexity of 10.

    Each concept is found in some sort of propositional statement. However, looking at the “structures” of logic, some statements are more useful than others at expressing how well the concept is truly understood.

    on the negative side, an atomistic statement (e.g. Values are important) has no significant value. Similarly, linear statements (e.g. More money causes more happiness which causes more laziness) are not valued, nor are circular statements (e.g. more work causes more play which causes more work).

    In contrast, what seems to be more effective is when there are two (or more) causal concepts influencing one resulting concept. That resulting concept is “concatenated” and is held to have a greater value because it is better understood.

    So, If we have a model where “More pay and More pep-talks cause More Productivity” we can say that there are three concepts (pay, pep-talks, productivity). So the complexity of the model is C=3. One of those concepts is concatenated (productivity) so it is well understood. However, the other two are not so well understood (where does all the pay come from? what constitutes a good pep-talk?). So, the Robustness of the model is R-0.33 (the result of one concatenated concept divided by three total concepts.

    Most policies and theories tend to be of low Complexity and low Rubustness. This provides one explanation why the don’t work as well as we would like. Using textual sources, it is possible to integrate multiple theories to rigorously create a theory that is more likely to be effective.

    For building understanding of policy in a group setting, we can bring out the propositional statements of the participants which reflect their mental maps. One important effect is that they may begin with many linear propositions… When those are combined, they may lead to the creation of concatenated concepts.

    The scores of any individual or sub-group will be less than the whole group. Further, as the group identifies overlaps between their perspectives, they gain new insights into how the world “works” (and the shared map has a higher score).

    They also gain new insights into how to work together more collaboratively because to the causal relationships.

    I hope that helps – Please feel free to send me an email and we can talk more. swallis [at] projectfast.org



  5. James Ivers Reply

    October 19, 2013 at 10:15 am

    Sounds interesting. Can someone give me a better idea of how the metrics are developed or set up?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>