Who Answers When AI Agents Get It Wrong? It Matters
Sam Bradon
Ritwik Singh
This article draws on insights from Who Answers for the Agent, a CTO Consulting podcast featuring Sam Bradon (director of platforms) and Ritwik Singh, founder of 36ARC.
Over the past two years, agentic AI has raced through three distinct phases faster than almost any enterprise technology before it. First came potential: impressive demos showing AI could draft, summarise, and respond. Then came proof of value: could it actually do the job reliably, with real customers, real data, and real consequences? Now we are entering the third and hardest phase: governance. The question enterprises face today is not whether AI can act, but who is accountable when it does.
Stuck Between Demo and Deployment
Many organisations remain caught in phase two, and the reasons are rarely about the models themselves. They are about context: fragmented technology stacks, inconsistent data quality, technical debt, and change management that has not kept pace with ambition. An agent that performs well in a controlled pilot often struggles when it meets messy inputs, unanticipated edge cases, and the sheer variability of live operations. Capability was never the bottleneck. Structure is. Agents need clarity on what they are allowed to do, when to escalate to a human, and a defensible record of what happened and why.
Regulators Are Naming the Gap
This is not a theoretical concern. In Australia, APRA’s targeted supervisory review of major banks, insurers, and superannuation trustees culminated in an industry-wide letter noting that adoption has outpaced governance, exposing institutions to cyber and operational risk. Tellingly, the letter also questioned whether boards can properly interrogate vendor claims, rather than simply accepting polished presentations at face value.
Around the same time, NSW Chief Justice Andrew Bell used one of Australian commercial law’s most prestigious lectures to make a pointed legal argument: a director’s protection under the business judgement rule only holds if the judgement was genuinely theirs. Adopting an AI-generated recommendation without independent scrutiny is not exercising judgement. It is transcription. Justice Lee’s remarks in the Star Entertainment Group proceedings reinforced the same principle: AI can help directors digest volume and complexity, but it cannot replace independent thought, and it cannot excuse disengagement from the information a director is required to control.
Guardrails Help, but They Are Not the Answer
Control planes, guardrails, and AI ‘control towers’ are genuinely useful, but they risk creating a false sense of resolution. Large language models do not offer faithful explanations of their own reasoning. What they produce after the fact is often plausible, not necessarily accurate. The more valuable concept is traceability: what data was retrieved, what policy version applied, which deterministic checks ran, and who signed off. Just as important is guarding against the quiet erosion of human oversight. When people are handed high volumes of AI-generated recommendations to approve, automation bias sets in. Approval becomes a reflex, not a judgement. A human in the loop who has stopped genuinely evaluating is not providing oversight; they are providing a signature.
Deterministic Structure, Probabilistic Judgement
The most workable model separates two things clearly: deterministic governance and probabilistic judgement. Rules engines, workflow enforcement, and control planes should define the boundaries, the ‘operating envelope,’ within which an LLM is permitted to interpret ambiguity, propose actions, and handle variation. None of this removes accountability. Organisations can buy tools that log activity, enforce constraints, and surface risk on a dashboard. What cannot be bought is the person willing to answer for where that boundary was drawn, and whether the resulting decision was one the institution should have made.
What Good Practice Looks Like
Singapore’s regulatory approach stands out as a constructive model, deliberately avoiding a rigid, anti-innovation stance in favour of a genuine framework. Its core principles: do not attempt to supervise every decision; instead tier risk and reserve human approval for the moments that truly matter; explicitly name automation bias as an organisational risk rather than pretending it does not exist; and establish clear lines of responsibility across the entire process, not just a kill switch bolted onto the model.
The Monday Morning Test
For enterprise leaders, the starting point is not technology. It is clarity of purpose. Before selecting tools, define the outcome you are trying to produce, stated in terms a person can be held to. From there, two requirements follow for any agentic system: it must be traceable, so its actions, inputs, and approvals can be reconstructed and defended; and it must be reliable, meaning it succeeds through a repeatable, defensible process rather than a track record of having been right so far.
Trust in agentic AI is not a product feature you purchase. It is an operating model you build, through explicit delegation, durable evidentiary records, meaningful escalation, and a named person prepared to answer for what the system does. Organisations that internalise this will move past the hype cycle. Those that do not will eventually discover, in the worst possible circumstances, exactly who was supposed to be answering for the agent all along.