Decision Audits Make for Better Decisions
The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.
By Robert Alan Young
July 10, 2023
Daniel Kahneman, the winner of the 2002 Nobel Prize in economics and author of Thinking, Fast and Slow, introduced insight into the two ways humans think. System 1 is faster, intuitive and more emotional, while System 2 is slower, more deliberative and logical. The first way is the most efficient in terms of time. In contrast, the second generates conclusions through analysis, more profound levels of reflection and a higher level of effectiveness in decision-making—but that takes longer. This dual-process approach is complementary to one another, balancing speed in some cases with complex and thorough analysis in others. The trick is in the balancing act. Decision-making involves the integration of experience and judgment with pertinent evidence.
The President’s Management Agenda cites the intention to advance data practices to support decisions and policymaking grounded in evidence about what works (The Biden-Harris Management Agenda Vision). In alignment with that intention, Acting FAA Administrator Billy Nolen suggests the FAA needs to transition to the next level of safety, one that is predictive, not just preventive, and rather than responding to incidents, needs to get better at predicting them through the careful analysis of data. But data itself does not produce decisions. Instead, the people involved in the decision-making process facilitate decisional outcomes. Such being the case, it makes sense to audit the decision-making process like any other management system. According to the ISO 19011:2018 guidelines for auditing a management system, an audit should be a systematic, independent and documented process for obtaining and evaluating evidence to assess the extent to which the criteria are fulfilled. A decision audit is a good tool for collecting and analyzing the evidence to determine the effectiveness of the decision-making process and where improvement opportunities exist.
Decision process audits vary in breadth and depth. For example, rapid assessment evaluates how teams generate decisions, say, the extent to which communications across the team play a role. Is the leader more directive, facilitative or consultative? Did everyone understand the acronyms? Did the team spend time chasing consensus succumbing to the pressure to achieve an unanimous agreement creating a groupthink decision? Was focus directed on shared interests or individual positions? Were the often undiscussable issues addressed or blushed over? Exploring the answers to questions like these can help improve weaknesses before moving to the next level of decision auditing.
The more comprehensive decision audits often apply qualitative data collection and analysis methods. In this regard, there is more rigor when sound methodology during the analysis and interpretation results in insights being considered more reliable. Methods such as participant interviews (anonymized) and evaluation of records and documents using latent content or thematic analysis will be used to tell more of the story when subjectively exploring the decision-making process. Observing the team and using a checklist with indicators of critical thinking surfacing as behaviors is a helpful data collection tool. We used such a tool at the FAA during a project early last year to introduce evidence-based management.
The decision audit, whether a rapid assessment or a more comprehensive examination, is separate from an after-action review where the focus is on the effectiveness of the outcome. The decision audit examines (1) process flow, (2) systems leveraged, (3) the extent to which evidence was introduced for consideration and (4) participant interaction leading up to the point when a decision is considered made.
Asking whether the problem or challenge was accurately and sufficiently framed and documenting which alternatives were considered is a foundational beginning. Recording all decisions in the minutes and the names of the people responsible for the actions fosters transparency and accountability. Meeting minutes are one activity that does require agreement by all parties involved. Once accepted, the meeting minutes serve as a record and are often a vital source of evidence. Many experts suggest a doubling down on the process flow for those cross-cutting decisions involving more substantial risk.
A recent decision audit at the FAA uncovered the natural tendency to revert to intuition when assessing the approach for revamping the strategic quarterly performance reviews framework. In place of taking the time to return to the originating documentation, where the collection of documents offered a comprehensive justification for the framework, nearly all respondents used their anecdotal experience to evaluate the efficiency and effectiveness of the framework. Anecdotes and their accompanying inferences are often accepted as evidence rather than continuing to seek information from other sources. In this scenario, referring to the original use case, the briefing slides and meeting minute notes highlighting the collaboration would have been more aligned with seeking richer evidence to inform conclusions and bolster the collective contributions towards the next phase of continual improvement.
The culture, and therefore the organization, would be well on their way to better decision-making if certain essential practices become commonplace. For example, asking genuine questions of one another, sharing all the relevant information, explaining the reasoning and intentions behind the questions, testing assumptions and inferences rather than relying on intuition and seeking relevant data and information from a broader range of sources. The hope is to counter cognitive biases that influence the dual-process System 1 tendencies to sway from rational decision-making. But, again, it’s not intentional; it’s human nature.
The evidence justifies focusing as much on the decision process as on the outcome. As Kahneman suggests, one of the tasks of System 2 is to overcome the impulses of System 1 and use deliberation instead of intuition when a scenario calls for more informed decision-making. Less so at the tactical level, where expert intuition is sufficient, but more at the strategic level, where complex and volatile challenges appear. When you use evidence drawn from a decision audit to improve the process, the decision-making is better and there is an improvement in the quality of the services being delivered to stakeholders.
Author: Dr. Robert Alan Young is the senior advisor for Security and Hazardous Materials Safety at the Federal Aviation Administration in Washington, DC. Bob can be reached at [email protected]
(No Ratings Yet)
Loading...
Decision Audits Make for Better Decisions
The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.
By Robert Alan Young
July 10, 2023
Daniel Kahneman, the winner of the 2002 Nobel Prize in economics and author of Thinking, Fast and Slow, introduced insight into the two ways humans think. System 1 is faster, intuitive and more emotional, while System 2 is slower, more deliberative and logical. The first way is the most efficient in terms of time. In contrast, the second generates conclusions through analysis, more profound levels of reflection and a higher level of effectiveness in decision-making—but that takes longer. This dual-process approach is complementary to one another, balancing speed in some cases with complex and thorough analysis in others. The trick is in the balancing act. Decision-making involves the integration of experience and judgment with pertinent evidence.
The President’s Management Agenda cites the intention to advance data practices to support decisions and policymaking grounded in evidence about what works (The Biden-Harris Management Agenda Vision). In alignment with that intention, Acting FAA Administrator Billy Nolen suggests the FAA needs to transition to the next level of safety, one that is predictive, not just preventive, and rather than responding to incidents, needs to get better at predicting them through the careful analysis of data. But data itself does not produce decisions. Instead, the people involved in the decision-making process facilitate decisional outcomes. Such being the case, it makes sense to audit the decision-making process like any other management system. According to the ISO 19011:2018 guidelines for auditing a management system, an audit should be a systematic, independent and documented process for obtaining and evaluating evidence to assess the extent to which the criteria are fulfilled. A decision audit is a good tool for collecting and analyzing the evidence to determine the effectiveness of the decision-making process and where improvement opportunities exist.
Decision process audits vary in breadth and depth. For example, rapid assessment evaluates how teams generate decisions, say, the extent to which communications across the team play a role. Is the leader more directive, facilitative or consultative? Did everyone understand the acronyms? Did the team spend time chasing consensus succumbing to the pressure to achieve an unanimous agreement creating a groupthink decision? Was focus directed on shared interests or individual positions? Were the often undiscussable issues addressed or blushed over? Exploring the answers to questions like these can help improve weaknesses before moving to the next level of decision auditing.
The more comprehensive decision audits often apply qualitative data collection and analysis methods. In this regard, there is more rigor when sound methodology during the analysis and interpretation results in insights being considered more reliable. Methods such as participant interviews (anonymized) and evaluation of records and documents using latent content or thematic analysis will be used to tell more of the story when subjectively exploring the decision-making process. Observing the team and using a checklist with indicators of critical thinking surfacing as behaviors is a helpful data collection tool. We used such a tool at the FAA during a project early last year to introduce evidence-based management.
The decision audit, whether a rapid assessment or a more comprehensive examination, is separate from an after-action review where the focus is on the effectiveness of the outcome. The decision audit examines (1) process flow, (2) systems leveraged, (3) the extent to which evidence was introduced for consideration and (4) participant interaction leading up to the point when a decision is considered made.
Asking whether the problem or challenge was accurately and sufficiently framed and documenting which alternatives were considered is a foundational beginning. Recording all decisions in the minutes and the names of the people responsible for the actions fosters transparency and accountability. Meeting minutes are one activity that does require agreement by all parties involved. Once accepted, the meeting minutes serve as a record and are often a vital source of evidence. Many experts suggest a doubling down on the process flow for those cross-cutting decisions involving more substantial risk.
A recent decision audit at the FAA uncovered the natural tendency to revert to intuition when assessing the approach for revamping the strategic quarterly performance reviews framework. In place of taking the time to return to the originating documentation, where the collection of documents offered a comprehensive justification for the framework, nearly all respondents used their anecdotal experience to evaluate the efficiency and effectiveness of the framework. Anecdotes and their accompanying inferences are often accepted as evidence rather than continuing to seek information from other sources. In this scenario, referring to the original use case, the briefing slides and meeting minute notes highlighting the collaboration would have been more aligned with seeking richer evidence to inform conclusions and bolster the collective contributions towards the next phase of continual improvement.
The culture, and therefore the organization, would be well on their way to better decision-making if certain essential practices become commonplace. For example, asking genuine questions of one another, sharing all the relevant information, explaining the reasoning and intentions behind the questions, testing assumptions and inferences rather than relying on intuition and seeking relevant data and information from a broader range of sources. The hope is to counter cognitive biases that influence the dual-process System 1 tendencies to sway from rational decision-making. But, again, it’s not intentional; it’s human nature.
The evidence justifies focusing as much on the decision process as on the outcome. As Kahneman suggests, one of the tasks of System 2 is to overcome the impulses of System 1 and use deliberation instead of intuition when a scenario calls for more informed decision-making. Less so at the tactical level, where expert intuition is sufficient, but more at the strategic level, where complex and volatile challenges appear. When you use evidence drawn from a decision audit to improve the process, the decision-making is better and there is an improvement in the quality of the services being delivered to stakeholders.
Author: Dr. Robert Alan Young is the senior advisor for Security and Hazardous Materials Safety at the Federal Aviation Administration in Washington, DC. Bob can be reached at [email protected]
Follow Us!