Win the AI game: How dependence on AI erodes human judgment

Executive Summary

AI adoption is accelerating across industries, and for most organizations, it is no longer a choice. The question is not whether to use AI; it is whether leaders understand what they are trading away as they do.

This piece identifies a compounding risk that is easy to miss precisely because it builds gradually: the more organizations defer to AI, the more they create the conditions for its failure. Model collapse erodes the quality of AI outputs when systems train on each other’s data. Cognitive atrophy erodes the human capacity to catch those errors. And competitive dynamics, a classic prisoner’s dilemma, lock firms into ever-deeper AI dependence even when the collective outcome worsens for everyone. These are not three separate problems. They are one feedback loop playing out simultaneously at the technical, human, and organizational level.

The firms that will outperform in the AI era are not those that automate the most. They are those that recognize AI for what it is, infrastructure, not intellect, and invest as deliberately in the human judgment layered on top of it as they do in technology itself. This means governing where AI operates without human challenge, training teams to interrogate outputs rather than defer to them, and holding the line on human accountability at consequential decision points. The calibration imperative is not a constraint on performance. It is the source of it.

Introduction

Artificial intelligence (AI) has rapidly become a trusted aid in decision-making across industries. From strategic planning to daily operations, executives are increasingly leaning on AI systems to analyze data and even to generate creative content. The real risk of AI is not that it fails.

It is that it succeeds just enough to make us stop thinking, and in doing so, sets in motion a compounding loop that degrades both the machines and the humans who depend on them. This piece argues that model collapse, cognitive atrophy, and competitive lock-in are not three separate concerns about AI over-reliance; they are three expressions of the same underlying dynamic: a feedback loop that quietly erodes the quality of decisions, ideas, and judgment across an entire system, technical, human, and organizational at once. Understanding that loop is the first step to breaking it.

At the center of this dynamic is what we call the AI echo chamber: a recursive loop where AI systems primarily learn from or respond to other AI systems rather than from reality. In such loops, predictive performance can degrade over time, as the models feed on each other’s outputs instead of fresh human input. The risk is that by outsourcing critical thinking to machines, we inadvertently amplify their limitations. AI models, after all, are powerful pattern-recognition engines without true understanding or sentience; when they begin to take cues from one another in a closed feedback cycle, errors and biases can compound while true signal is lost.

This article explores this phenomenon and its far-reaching implications, from technical degeneration of AI performance and strategic traps for businesses, to the cognitive and ethical debts we accumulate as human judgment yields to algorithmic convenience. We will illustrate the dynamics with a duopoly game theory scenario, delve into the psychological drivers of over-reliance (why humans are drawn to “easy” thinking), and draw on both modern research and wisdom (including Dietrich Bonhoeffer’s warnings about intellectual passivity) to understand the stakes. The goal is a thought-provoking perspective for leaders: to recognize the subtle dangers of AI over-reliance and strike a wiser balance between machine efficiency and human critical thought.

The recursive AI-to-AI loop – When AI feeds on AI

One key danger of ubiquitous AI adoption is the creation of self-referential loops: AI systems drawing input from other AI outputs. In essence, AI starts feeding on AI. This can happen unintentionally, for example, when one model’s generated content becomes part of the training data or when algorithmic decisions are made in response to other algorithms. The problem is that without fresh human reality checks, these systems risk drifting away from ground truth. Recent research has even given this a name: “model collapse.” In a 2024 Nature article, Shumailov et al. demonstrated that if generative models (like large language models or image generators) are trained predominantly on data produced by other AIs, they begin to “forget” the true underlying data distribution (Shumailov et al., 2024). In their experiments, the AI’s outputs became progressively narrower and less accurate, with the rich variability (“the tails” of the distribution) disappearing over successive generations of training (Shumailov et al., 2024). In plain terms, when an AI learns mostly from AI-generated data, it inherits the previous models’ mistakes and loses touch with reality. Errors get reinforced rather than corrected, and over time the system’s predictions converge to a bland, misleading average.

This degeneration is somewhat analogous to making a copy of a copy of a copy: eventually, the fidelity drops. An AI system lacks the true common-sense reasoning or adaptive imagination of a human; it cannot invent genuinely new insights beyond the patterns it was trained on. So, if those patterns become polluted by artificial outputs, which often carry subtle errors or omissions, the AI has no innate compass to course correct. Garbage in, garbage out, but here the “garbage” is coming from another AI. The result is a feedback loop of declining quality. The predictive performance degrades, sometimes in ways that may not be immediately obvious until a critical failure occurs.

AI practitioners are already grappling with this directly. For example, developers of large language models (LLMs) like GPT-5 are wary of the internet becoming flooded with AI-generated text. If future models train on a web where a high percentage of content was machine-written, they risk ingesting second-hand data. As Shumailov et al. warn, “indiscriminate use of model-generated content in training causes irreversible defects in the resulting models”, leading to irrecoverable loss of knowledge from the original human data (Shumailov et al., 2024). In short, the more AI learns from AI, the dumber it could get.

An AI arms race in  business: A game-theoretic perspective

Why would we ever allow AI systems to primarily talk to each other in the first place? Often, it’s an unintended consequence of competitive dynamics.

Consider an isolated market with two rival firms. Each firm faces a strategic choice: adopt advanced AI decision systems or stick to human-led processes. It might seem preferable if both stayed human-driven, avoiding the costs and uncertainties of AI. However, game theory shows that each firm has a strong incentive to adopt AI if the other might do so. In fact, using AI can become a dominant strategy (the best response regardless of the competitor’s move). If Firm A alone adopts AI, say for pricing, investment, or trading decisions, while Firm B doesn’t, A could gain an edge by reacting faster or finding patterns B misses. Firm B would suffer a competitive disadvantage. So B will feel compelled to also adopt AI to keep up. Conversely, if B has AI and A does not, A loses out. Thus, the likely outcome is both adopt AI, even if in hindsight, they’d prefer a world of neither. This outcome is a classic Nash equilibrium: no firm can unilaterally deviate (drop their AI) without harming their payoff. In essence, it becomes an AI arms race, once one side deploys it, the other must as well.

While this equilibrium might maximize short-term performance parity, it sets the stage for the recursive AI-to-AI interactions we described. Now both firms’ decisions are largely driven by algorithms reacting to algorithms. The competitive market becomes a fast-paced conversation of machines. This is not hypothetical; it’s already evident in domains like high-frequency trading in finance. By the late 2010s, algorithms executed the majority of trades in many stock markets, operating at speeds and complexities beyond direct human control (Frazier, 2024). Traders adopted AI-driven models because if even one major player did, others had to follow suit or be outpaced (Frazier, 2024). The result has been instances where these algorithms began effectively “talking” to each other and amplifying each other’s mistakes. A famous example is the May 6, 2010 “Flash Crash.” On what seemed like a normal trading day, a few algorithms misread routine market signals and initiated a rapid sell-off. Their automated selling triggered other algorithms to also start selling in a cascade. Within minutes, the U.S. stock market plummeted nearly 1,000 points, erasing about $1 trillion in value, before human intervention helped correct it (Frazier, 2024). Investigators later noted that a minor event was magnified into a crisis because so many AIs were responding to each other in uniform ways. As SEC Chair Gary Gensler warned, when many actors rely on “the same underlying model or data,” a small glitch can “cause a slight downturn to become a rapid collapse” (Frazier, 2024).

In our two-firm scenario, imagine both companies use AI pricing algorithms that continuously undercut each other by pennies to win market share. Left unchecked, they might engage in an endless loop of price reductions (or oscillations) at machine speed, yielding volatile outcomes neither CEO intended. Alternatively, the algorithms might learn that price wars hurt both, and without explicit collusion by humans, they could tacitly stabilize at high prices, effectively creating algorithmic collusion. Either way, human managers cede a degree of control and visibility. The Nash equilibrium of both using AI ensures the firms stay evenly matched, but it also leaves the battlefield to the machines, whose rapid interactions may produce weird, suboptimal, or even dangerous dynamics. Executives thus find themselves in a paradox: compelled to adopt AI to remain competitive, yet collectively ending up in a situation where AI-centric interactions diminish the quality of outcomes. The strategic takeaway for leaders is that adopting AI is not just a straightforward win: it can introduce systemic risks that no single firm acting alone can mitigate. It calls for new strategies to manage the recursive effects of AI-on-AI engagement. Critically, this is a prisoner’s dilemma: no single firm can solve it through restraint alone. The implication for executives is that industry-wide coordination, or proactive regulatory engagement, may be the only rational escape from a race that collectively degrades outcomes for all players.

Cognitive offloading: Why humans are drawn to “easy” thinking

The prevalence of these AI loops is not solely a technological or competitive artifact; it’s also driven by human psychology. Simply put, humans tend to take the path of least cognitive resistance (Risko & Gilbert, 2016). According to Zipf’s principle of least effort, people tend to organize actions to minimize the total work required to solve problems.

If a machine will do the thinking, we often let it. Behavioral science and psychology have long documented our inclination toward mental shortcuts. Nobel laureate Daniel Kahneman famously described our thought processes as two systems: System 1, which is fast, automatic, and energy-efficient (our gut instincts and quick reactions), and System 2, which is slow, effortful, and analytical (our deliberate reasoning and critical thinking) (Simons Summaries, n.d.). Crucially, System 2, though capable of deeper analysis, is “lazy” or cost-averse (Simons Summaries, n.d.). Unless we absolutely must engage in hard thinking, we often default to the quick intuitions of System 1.

AI tools present the ultimate temptation for our lazy System 2. Why struggle over a complex problem or lengthy analysis when an AI assistant can give a quick answer or recommendation? This is known as cognitive offloading: offloading mental tasks to an external agent (like relying on GPS for navigation rather than remembering the route). In moderation, such offloading is efficient. But excessive offloading to AI can lead to what some researchers are calling “cognitive debt.” (Chow, 2025),  (EDUCAUSE Review, 2025),  (Gerlich, 2025). It’s akin to technical debt in software: you gain a short-term benefit (fast answers, saved effort) at the cost of long-term erosion of your own capabilities. A group of scientists recently warned that the persuasive convenience of AI chatbots is exacerbating our mental laziness and loss of critical thinking (Port, 2024). In other words, when answers are a click or voice-command away, our incentive to think deeply or question diminishes further (Simons Summaries, n.d.).

This trend is supported by emerging data. One study in an educational setting found that students with a higher “Need for Cognition” (a trait describing how much a person enjoys effortful thinking) were less likely to blindly follow AI suggestions, they scrutinized and verified the AI’s recommendations more often (Pitts et al., 2025). Those low in this trait (i.e., more cognitively apathetic) tended to over-rely on the AI, accepting its answers even when incorrect. The danger is clear: if we don’t enjoy or at least willingly engage in effortful thought, we become prime candidates to trust AI outputs uncritically.

Over-reliance on AI thus taps into a form of cognitive laziness: our brains happily avoid exertion and treat AI’s output as a mental shortcut. It feels like using System 1 by proxy: the AI gives an instantaneous answer with an air of authority, sparing us the discomfort of activating System 2. The irony is that while this feels efficient, it may be making us intellectually weaker. We fail to practice the skill of reasoning through problems, just as muscle atrophies from disuse. And unlike delegating to a human expert (where at least a chain of human reasoning is behind the advice), delegating to an AI means trusting a process that we might not fully understand or that might be statistically prone to error, even if it sounds confident. In a sense, we risk outsourcing our judgment and common sense.

From an executive perspective, this human tendency means employees and decision-makers might too readily defer to AI: “The model says so, so that must be the right answer.” It takes conscious effort and a supportive culture to push back and ask critical questions that might catch a flaw in the AI’s suggestion. If that culture and effort are lacking, an organization can drift into automation complacency. People become button-pushers executing the AI’s outputs, rather than thinkers, fast to act but slow to truly reflect.

The costs of outsourcing thinking: Technical, managerial, and cognitive debt

If the previous section addressed the individual psychology of over-reliance, this section examines what that costs at scale, across teams, organizations, and systems. One immediate cost is the aforementioned cognitive debt at the individual level; the loss of skill and mental acuity. A striking new study by researchers at MIT’s Media Lab (2025) provides early evidence of this effect. In the study, participants were asked to write essays, and one group was allowed to use an AI (ChatGPT) to assist, while another had only a search engine, and a third had no tools. The results were eye-opening: those using the AI showed significantly less brain activity on EEG scans than those writing on their own (Chayka, 2025). The AI-assisted writers’ brain patterns had weaker connectivity, particularly in regions tied to creativity and memory (Chayka, 2025). Many of them didn’t feel ownership of the output, tellingly, 80% of the AI-assisted group could not quote a single full sentence from the essay they “wrote” (because, in effect, the AI wrote it) (Chayka, 2025). Kosmyna et al. dubbed this the “cognitive cost” of relying on AI (Chayka, 2025). Essentially, when you let the model take over, your brain “tunes out” sooner. Over repeated use, the AI-users in the study underperformed on both cognitive and creative measures compared to those who had been writing unaided (Chayka, 2025).

Beyond neural activity, there’s the issue of homogenization of output. The same MIT study found that when people use AI, their solutions start to look the same. The diversity of thinking erodes. For instance, in the MIT experiment, the essays from students using ChatGPT converged on similar arguments and phrasings, even for open-ended questions; the AI essentially pulled them all toward a common mean (Chayka, 2025). One researcher noted that with the LLM’s help, “you have no divergent opinions being generated… Average everything everywhere all at once”, as if the AI collapses unique perspectives into a bland consensus (Chayka, 2025). Likewise, a separate Cornell study observed that when writers from different cultural backgrounds used an AI auto-complete, their responses became more similar and aligned to Western norms (e.g. many different people suddenly claim their favorite food is pizza, in the AI-assisted group) (Chayka, 2025). In short, AI’s tendency to replicate patterns makes our outputs more average (Chayka, 2025). A New Yorker column encapsulated it well: “A.I. is a technology of averages”; it spots patterns across vast data and reproduces answers that lean toward consensus, clichés, and banalities, ultimately pulling our thinking to that average if we lean on it (Chayka, 2025). For businesses, this raises a worrying thought: over-reliance on AI could stifle the outlier ideas and creative sparks that often drive innovation. If every strategy memo or marketing copy starts to sound uniformly “AI-generic,” the company’s distinctive voice and competitive edge may dull.

There’s also an organizational knowledge debt. Employees who stop honing their expertise because “the AI will figure it out” accumulate managerial and technical debt for the firm. Imagine a future scenario where your supply chain AI has optimized logistics for years, but one day it breaks or encounters a scenario outside its training. Do any humans still understand the intricacies well enough to step in? Or consider strategic planning: if executives come to lean on AI analyses without independently understanding the market drivers, the organization may lose the human strategic intuition that’s vital in crises or novel situations. In essence, by always choosing the fast, AI-powered answer (analogous to Kahneman’s System 1 outputs), a company might neglect the slower, deeper thinking (System 2 deliberation) that is necessary for long-term evolution and breakthrough ideas. This is reminiscent of how relying only on short-term metrics and neglecting long-term R&D is a debt that eventually comes due.

From a technical standpoint, blindly trusting AI outputs can lead to error cascades. We already discussed how AI models can reinforce errors from each other; at the user end, there’s something known as automation bias: people accepting automated suggestions too readily and failing to monitor for mistakes. This has been observed in domains like aviation and medicine, where over-trusting an autopilot or diagnostic AI can have dire consequences if the automation is wrong. Thus, the debt here is a safety and quality debt: it accumulates quietly as minor lapses and over-reliance but can manifest dramatically when the AI makes a significant misjudgment that no human is prepared (or skilled enough) to catch and correct in time.

Bonhoeffer’s warning: The decline of critical discourse

Long before modern AI, the theologian and anti-Nazi dissident Dietrich Bonhoeffer reflected on how societies lose their intellectual rigor. In his 1943 letters from prison, he wrote about a form of stupidity that was not a matter of low IQ, but of moral and intellectual passivity, a surrender of independent thought so complete that the person becomes immune to reason. He observed that this kind of stupidity was “a more dangerous enemy of the good than malice,” because the passive mind, utterly self-satisfied, simply dismisses opposing facts as inconsequential (Bonhoeffer, 1953; Sarkar, 2023). In his context, he was describing how a populace under propaganda could surrender its critical faculties and follow harmful ideologies without question.

The analogy is worth mapping precisely. In Bonhoeffer’s context, the coercive power was a political regime that rewarded conformity and punished dissent. In the AI context, the “power” is subtler: it is the perceived authority of data-driven outputs, the social pressure to defer to what “the model says,” and the institutional incentives that reward speed over scrutiny. The mechanism of passivity is the same, external authority substitutes for internal judgment, even if the stakes and the coercion are of a different order.

The organizational risk does not stop at the office door. Content farms generating AI-written news, social media flooded with bot-generated posts, and machine-curated information feeds are already shaping the opinions of customers, regulators, and markets (Inter-Parliamentary Union, n.d.). For executives, this is not an abstract concern: a workforce that has grown intellectually passive internally is poorly equipped to recognize or respond to AI-driven narrative manipulation externally (Rigotti & Fosch-Villaronga, 2024). Bonhoeffer’s warning applies at the institutional level too, an organization that stops questioning its own data-driven conclusions becomes equally unlikely to question the data-driven conclusions being fed to it from outside (Inter-Parliamentary Union, n.d.).

The remedy Bonhoeffer points to, valuing independent thought and the courage to question, is precisely what leaders must cultivate in the AI era. Encouraging teams to ask “Why?” even after the model has spoken, treating AI outputs as propositions to evaluate rather than verdicts to execute, and rewarding the analyst who pushes back: these are not inefficiencies. They are the guardrails against the learned helplessness that Bonhoeffer warned leaves us defenseless against folly (Sarkar, 2023).

Reclaiming autonomy: Ethical and strategic guardrails

It becomes clear that mitigating these risks isn’t just a technical task, it’s an ethical and leadership challenge. One core principle emerging from AI ethics discussions (such as those at Oxford) is the importance of preserving human autonomy and oversight. In high-stakes domains especially, AI should support human decision-making, not replace it outright. The Inter-Parliamentary Union, for example, in its guidelines for AI, emphasizes that humans “should be free to make decisions about their lives without interference, coercion or manipulation,” and that use of AI must be designed so as not to undermine that freedom (Inter-Parliamentary Union, n.d.). In a corporate context, this translates to ensuring AI tools remain advisors rather than autonomous rulers. An executive decision-support AI might crunch numbers and highlight patterns, but the final call, with all its moral and strategic dimensions, should ideally be made by a human agent who can be held accountable. This maintains a feedback loop to human values and common sense, which purely data-driven systems can lack.

We must remember that AI’s intelligence is narrow and historical. By design, most AI models (from predictive analytics to large language models) learn from historical data. They excel at pattern recognition within the domain of what’s happened before. But they struggle with novelty, situations that deviate from past patterns, and they have no innate grasp of causality, context, or ethical principles unless explicitly trained or constrained, and even then, in a brittle way. Any AI ethics course might remind us that no matter how sophisticated an AI’s output, it fundamentally “has only seen the past.” It cannot truly imagine futures that break precedent, nor can it weigh right vs. wrong the way a human conscience or legal framework would. Therefore, an over-reliance on AI can lead to reinforcing past biases and blind spots. For example, if an AI hiring tool is trained on past successful applicants, it might perpetuate demographic biases present in that historical data, unless humans intervene to correct it. There have been real-world cases of this, where AI recruitment systems had to be scrapped after it was found they unfairly disadvantaged certain groups because the training data reflected past hiring imbalances (Rigotti & Fosch-Villaronga, 2024),  (Dastin, 2018). Ethical use of AI requires continuous human oversight to catch these issues and inject judgment where the machine cannot.

Another ethical concern with recursive AI loops is accountability. When algorithms trade with algorithms, or when AI recommends one managerial decision after another, who is responsible if something goes wrong? It’s easy for accountability to become diffuse: the human operators might shrug, “the AI recommended it and it usually knows best.” This is unacceptable, especially when decisions affect stakeholders’ lives. Maintaining human-in-the-loop governance is critical. Many AI governance frameworks insist on mechanisms for human review, override, or auditing of AI decisions (Inter-Parliamentary Union, n.d.). For executives, a practical step is to set boundaries on AI autonomy. For instance, use AI to flag fraud cases, but let humans review before action; or use AI in medical diagnosis as a second opinion, not the sole voice. By requiring that important decisions get a “human stamp of approval,” organizations protect against machine errors and ensure humans stay mentally engaged with the decision process. This can prevent the skill fade we discussed and keep responsibility where it belongs.

Finally, leaders should weigh the concept of resilience. A system overly dependent on AI is brittle if the AI fails or is compromised. True resilience comes from hybrid human-AI systems where each compensates for the other’s weaknesses. Humans offer contextual understanding, ethical judgment, and creative thinking; AI offers speed, scale, and consistency on well-defined tasks. Together, if managed well, they outperform either alone. But achieving this means actively managing that interface, training staff to work with AI critically, setting up audit trails for AI decisions, and cultivating a mindset that treats AI output as input, not gospel. Ethically, it also means being transparent: ensuring that when AI is used (say, algorithmic pricing or credit scoring), the people affected are aware and some form of explanation or recourse is available. Reduced human autonomy in an AI-driven scenario is not an inevitable fate; it’s a choice we make by either handing over the keys fully or keeping a hand on the steering wheel.

It would be incomplete to suggest that the research community has not responded to these risks. Practitioners are already developing mitigations: synthetic data filtering to reduce model collapse (Shumailov et al., 2024), Reinforcement Learning from Human Feedback (RLHF) to better align outputs with human values, (Christiano et al., 2017; Bai et al., 2022; Zhao et al., 2024) constitutional AI approaches that embed ethical constraints directly into model behavior (Bai et al., 2022), and watermarking techniques designed to identify AI-generated content before it re-enters training pipelines (Zhao et al., 2024). These are meaningful advances. The problem is not that solutions are absent, it is that adoption is uneven, implementation is technically demanding, and none of these mechanisms address the human and organizational dimensions of the echo chamber. A model that is technically well-governed can still operate inside an organization that has lost the capacity to question it. The technical guardrails and the human ones must be built in parallel; one without the other is insufficient.

Diagnosing your exposure: The AI dependency risk matrix

Understanding the risks of AI over-reliance is one thing. Knowing where your organization actually stands is another. The matrix below offers a practical framework for that self-diagnosis, mapping exposure across two dimensions that, in our experience, most organizations have never explicitly assessed together: the degree to which AI operates without meaningful human challenge, and the organization’s remaining capacity to detect and correct its errors. The interaction of these two variables determines not just your current risk profile, but the trajectory you are on.

Why these two dimensions specifically? Because the most dangerous failure mode is not AI autonomy alone, many high-performing organizations run AI systems with significant independence and manage it well. The danger emerges when autonomy and eroded human capacity combine. An organization can move from Zone 2 to Zone 4 not through a single decision, but through a gradual drift: oversight processes that quietly fall away, expertise that atrophies through disuse, and a culture that increasingly treats the model’s output as the end of the conversation rather than the beginning of one.

Three patterns of drift are worth watching for, each one represents a distinct path toward Zone 4: