Unchecked AI Autonomy Poses Gravest Risk

If you’ve ever ridden in a self-driving Uber through downtown Los Angeles, you might recall that peculiar uncertainty that creeps in when there's no driver and no conversation—just a silent car interpreting the world around it. The ride seems normal until the vehicle misreads a shadow or brakes abruptly for no real threat. In that instant, you glimpse the core issue with autonomy: it doesn't panic when it should. That gap between its confidence and its judgment is where trust is either built or broken. Much of today’s enterprise AI feels strikingly similar. It is capable yet not confident, efficient yet not empathetic. That’s why the critical factor in any successful deployment is no longer raw computing power, but trust.

The MLQ State of AI in Business 2025 [PDF] report puts a sharp figure on this: 95% of early AI pilots fail to deliver measurable ROI. This isn't due to weak technology, but because it’s applied to the wrong problems. The pattern repeats across industries. Leaders grow uneasy when they can’t verify the output’s accuracy; teams doubt their dashboards; customers lose patience when an interaction feels robotic rather than supportive. Anyone who’s been locked out of their bank account while an automated recovery system insists their answers are wrong knows how swiftly confidence evaporates.

Klarna remains the most publicized example of large-scale automation at work. The company has halved its workforce since 2022 and reports its internal AI now does the work of 853 full-time roles, up from 700 earlier this year. Revenue has surged 108%, and average employee compensation has risen 60%, partly funded by these operational gains. Yet the picture is more complex. Klarna still posted a $95 million quarterly loss, and its CEO has warned of further staff reductions. This shows that automation alone doesn’t guarantee stability. Without proper accountability and structure, the user experience breaks down long before the AI does. As Jason Roos, CEO of CCaaS provider Cirrus, states, “Any transformation that erodes confidence, internally or externally, carries a cost you can’t ignore. It can leave you worse off.”

We’ve already seen what happens when autonomy outpaces accountability. The UK’s Department for Work and Pensions used an algorithm that wrongly flagged around 200,000 legitimate housing-benefit claims as potentially fraudulent. The problem wasn’t the technology—it was the lack of clear ownership over its decisions. When an automated system suspends the wrong account, rejects a valid claim, or causes unnecessary alarm, the question isn’t just “why did the model fail?” It’s “who owns the outcome?” Without an answer, trust becomes fragile.

“The missing step is always readiness,” says Roos. “If the process, data, and guardrails aren’t in place, autonomy doesn’t boost performance—it magnifies weaknesses. Accountability must come first. Start with the desired outcome, identify wasted effort, verify your readiness and governance, and only then automate. Skip those steps, and accountability vanishes as quickly as the efficiency gains appear.”

Part of the problem is an obsession with scale without the foundation to sustain it. Many organizations rush to deploy autonomous agents that can act decisively, yet few consider what happens when those actions stray beyond expected boundaries. The Edelman Trust Barometer [PDF] shows a steady five-year decline in public trust in AI. A joint KPMG and University of Melbourne study found workers prefer more human involvement in nearly half the tasks examined. These findings underscore a simple truth: trust rarely comes from pushing models harder. It comes from people understanding how decisions are made and from governance that acts less like a brake and more like a steering wheel.

The same dynamic appears on the customer side. PwC’s trust research reveals a wide gap between perception and reality: most executives believe customers trust their organization, but only a minority of customers agree. Other surveys show transparency helps close this gap, with most consumers wanting clear disclosure when AI is used in service interactions. Without that clarity, people don’t feel reassured—they feel misled, straining the relationship. Companies that communicate openly about their AI use aren’t just protecting trust; they’re normalizing the idea that technology and human support can coexist.

Some confusion stems from the term “agentic AI” itself. Much of the market treats it as something unpredictable or self-directing, when in reality it’s workflow automation enhanced with reasoning and recall. It’s a structured way for systems to make limited decisions within human-designed parameters. Successful, scalable deployments all follow the same sequence: they start with the outcome to improve, identify unnecessary effort in the workflow, assess system and team readiness for autonomy, and only then select the technology. Reversing this order doesn’t speed things up—it just leads to faster mistakes. As Roos notes, AI should expand human judgment, not replace it.

All this points to a broader truth. Every wave of automation eventually becomes a social question, not just a technical one. Amazon built its dominance through operational consistency, but also through the confidence that your package would arrive. When that confidence slips, customers leave. AI follows the same pattern. You can deploy sophisticated, self-correcting systems, but if a customer feels tricked or misled at any point, trust shatters. Internally, the same pressures apply. The KPMG global study [PDF] highlights how quickly employees disengage when they don’t understand how decisions are made or who is accountable. Without that clarity, adoption stalls.

As agentic systems take on more conversational roles, the emotional dimension grows even more significant. Early reviews of autonomous chat interactions show people now judge their experience not only by whether they were helped, but by whether the interaction felt attentive and respectful. A customer who feels dismissed rarely keeps their frustration to themselves. The emotional tone of AI is becoming a genuine operational factor, and systems that fail to meet this expectation risk becoming liabilities.

The hard truth is that technology will always move faster than people’s instinctive comfort with it. Trust will inevitably lag behind innovation. This isn’t an argument against progress, but a call for maturity. Every AI leader should ask: would I trust this system with my own data? Can I explain its last decision in plain language? Who steps in when something goes wrong? If those answers are unclear, the organization isn’t leading a transformation—it’s preparing an apology.

Roos puts it simply: “Agentic AI isn’t the concern. Unaccountable AI is.”

When trust goes, adoption goes with it, and the project that seemed transformative becomes another entry in the 95% failure rate. Autonomy isn’t the enemy. Forgetting who is responsible is. The organizations that keep a human hand on the wheel will be the ones still in control when the self-driving hype eventually fades.