When Artificial Intelligence Misses the System: A Cautionary Tale for Public Governance

The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.

By Mauricio Covarrubias
July 7, 2025

Artificial intelligence (AI) has rapidly become a cornerstone of digital transformation in the public sector. Governments are embracing algorithmic tools to manage complexity, increase responsiveness and reduce administrative burdens. From welfare automation to predictive policing, AI is touted as a solution to institutional inefficiencies. But this enthusiasm often comes with a critical oversight: the belief that AI is a neutral, technical fix for complex governance challenges.

This assumption reflects what systems theorist Donella Meadows describes in her work Thinking in Systems as a linear and mechanistic mindset—a way of thinking that seeks to optimize outputs without examining the broader interconnections that shape institutional performance and citizen outcomes. When AI is deployed without systemic awareness, it can not only replicate but amplify governance failures.

The Perils of Fragmented AI Deployments

Real-world cases across the globe highlight what happens when AI is introduced without attending to interdependencies, social context and long-term effects:

• In the Netherlands, an algorithmic fraud detection system known as SyRI was used to identify potential welfare fraud. However, the system disproportionately targeted low-income and immigrant neighborhoods, eventually prompting a court to declare it a violation of human rights. Scholars like Luke Munn have critiqued such systems in his book Ferocious Logics, arguing that algorithmic interventions often inherit and reinforce existing social biases when not critically examined.
• In Australia, the Robodebt scheme used automated data-matching to identify overpayments in the welfare system. Thousands of recipients received erroneous debt notices based on flawed calculations. The Royal Commission into the Robodebt Scheme in 2023 concluded that the program was not only deeply flawed but “cruel,” highlighting severe administrative failures and a lack of human oversight. Virginia Eubanks, in Automating Inequality, explores this case to illustrate how automated systems often punish the poor under the guise of efficiency.
• In the United States, predictive policing tools like PredPol have come under fire for reproducing racial biases embedded in historical arrest data. Ruha Benjamin addresses this in Race After Technology, calling attention to the dangers of what she terms the “New Jim Code”—the automation of racial discrimination under a façade of neutrality. Legal scholar Andrew Ferguson, in The Rise of Big Data Policing, also critiques these systems for lacking transparency and exacerbating surveillance in over-policed communities.

These are not simply “bad algorithms”—they are bad systems interventions. The failure lies in the poor framing of the public problems being addressed and in the absence of systemic awareness in the design and deployment of the technology.

The Problem of Framing and Boundary Critique

One persistent blind spot is how public problems are framed prior to the application of AI. Geoff Mulgan, in his book Big Mind, argues that governments tend to favor AI applications that discretize problems into algorithm-friendly formats—neglecting deeper, interconnected dynamics and social complexities.

This leads to what systems theorist Gerald Midgley calls a boundary critique failure—a lack of critical reflection on what (or who) is included or excluded from the system under consideration. In Systemic Intervention, Midgley highlights how narrow problem framing can undermine the effectiveness and fairness of interventions.

The Fallacy of Technological Neutrality

Another dangerous misconception is the belief in algorithmic neutrality. Frank Pasquale, in The Black Box Society, exposes how data-driven systems are anything but neutral. Algorithms are built on institutional goals, historical data and implicit biases—all of which reflect human decisions and values

Similarly, Virginia Eubanks underscores that automated systems often encode inequality, especially when applied to vulnerable populations. When deployed without systemic reflection, AI becomes a tool of what she calls “digital poorhouses”—systems that entrench disadvantage rather than promote justice.

Learning Deficits and the Risk of Path Dependence

Perhaps the most alarming risk is the erosion of institutional learning. Once AI systems are embedded, their outputs can become unchallenged reference points. Paul Pierson describes this in political science as path dependence—the idea that institutional choices create self-reinforcing trajectories that become difficult to reverse

Otto Scharmer, in The Essentials of Theory U, warns of the tendency for institutions to operate in “downloading mode”—repeating past behaviors and assumptions even when facing new conditions.  Without feedback loops and critical reflection, AI does not enhance institutional intelligence; it may automate dysfunction.

Toward Systemic Intelligence in AI Governance

What’s the alternative? As public leaders, we must reframe AI not as a “fix” but as a component in a broader system—one that involves people, institutions, values and power. This means asking different questions:

• What are the interdependencies involved in the issue?
• Who is affected—and who defines the problem?
• How can feedback and adaptation be embedded in the system?
• Where are the leverage points for meaningful change?

The goal is not to replace human decision-making but to enhance collective intelligence. AI must be aligned with public purpose, and that requires more than data or algorithms—it requires systemic thinking.

Governance is not an engineering problem. It is relational, adaptive and deeply human. AI can help us govern more wisely—but only if we build it on foundations of transparency, inclusion and systemic awareness.

---

Author: Mauricio Covarrubias is Professor at the National Institute of Public Administration in Mexico. He holds a Ph.D. from the National Autonomous University of Mexico (UNAM) and completed a postdoctoral fellowship in Government and Public Policy at the University of New Mexico in Albuquerque. He is co-founder of the International Academy of Political-Administrative Sciences (IAPAS). He can be reached at [email protected] and followed on X (formerly Twitter) @OMCovarrubias. 

(7 votes, average: 5.00 out of 5)
Loading...