Why AI Governance Is Really an Operations Problem

Jan Esman

Dale Rogers

This article draws on insights from Swiss Cheese and Bow Ties, Novel Models of AI Risk, a CTO Consulting podcast featuring Jan Esman, CTO Consulting’s Head of Enterprise Strategy, and Dale Rogers, Atturra’s Senior Consultant, Strategy and Service Design.

As enterprise AI moves from experimentation into mission-critical operations, many organisations are making a dangerous assumption: that governance begins and ends with policies, ethics frameworks, and model alignment.

Although these elements matter, they do not manage AI risk once a model is deployed into production. They establish intent, define acceptable use, and provide guardrails for responsible adoption, but they do not replace operational assurance.

The next phase of AI maturity requires organisations to think less like software developers and more like operators of critical infrastructure.

This was the central message of a recent CTO Consulting podcast exploring what digital transformation leaders can learn from industries that have spent decades managing complex operational risk.

The answer lies in an unlikely place: the Swiss cheese model.

Originally developed to understand failures in aviation, healthcare, and industrial safety, the Swiss cheese model recognises that every control contains weaknesses. Policies have gaps. Technical controls have limitations. Human oversight is imperfect. AI alignment is never absolute. Failure occurs only when those weaknesses align, allowing an error to pass through every layer of defence.

The lesson for AI governance is profound.

Many organisations continue to view governance as a compliance activity. They establish responsible AI principles, publish governance policies, and conduct ethics reviews before deployment. Although these activities are essential, they represent only the outermost layer of defence.

Operational governance begins after deployment.

Unlike traditional enterprise applications, AI systems do not remain static. They operate in dynamic environments, respond to changing inputs, interact with unpredictable users, and produce probabilistic outputs rather than deterministic ones. Governance, therefore, cannot be treated as a project milestone. It must become an ongoing operational capability.

This shift fundamentally changes how organisations should think about assurance.

Traditional ICT programmes measure success by delivering projects on time, within budget, and to specification. AI programmes require a different measure of success: maintaining reliable performance over time while continuously identifying and managing emerging risks.

That demands capabilities many organisations have yet to develop.

Continuous monitoring, output validation, telemetry, independent verification, model performance sampling, and operational testing become as important as model selection or prompt engineering. Rather than asking whether an AI system passed user acceptance testing, leaders should ask whether it continues to perform safely under real-world conditions months after deployment.

Equally important is recognising that AI failures rarely resemble the dramatic scenarios often portrayed in the media.

The greatest organisational risks are frequently subtle.

A recommendation engine develops a slight bias. A decision-support tool begins overlooking a particular customer cohort. An automated assessment process gradually deviates from expected outcomes. Individually, these errors appear insignificant. Collectively, they can reshape business decisions, customer experiences, and regulatory outcomes before anyone notices.

The danger is compounded by increasing user confidence.

As AI consistently delivers useful results, people naturally lower their scrutiny. Human reviewers become less likely to question outputs, allowing small inaccuracies to propagate unchecked. This phenomenon, often described as automation complacency, has long been recognised in aviation and industrial operations. AI introduces the same behavioural challenge into knowledge work.

Managing these risks requires more than preventative controls.

Another concept discussed during the podcast, Bow Tie analysis, offers an equally valuable perspective. Widely adopted across high-consequence industries, Bow Tie analysis distinguishes between controls that prevent incidents and controls that reduce consequences once incidents occur.

This distinction exposes one of the biggest weaknesses in current AI governance programmes.

Most organisations invest heavily in prevention. They fine-tune models, develop responsible AI frameworks, establish governance committees, and implement prompt engineering standards.

Far fewer prepare for failure.

Who has authority to turn off an AI service? What operational thresholds trigger intervention? How quickly can an organisation identify harmful outputs? Are legal, communications, cybersecurity, and executive response plans already established?

These questions become increasingly important as AI begins supporting regulatory decisions, healthcare, financial services, infrastructure, and public administration.

Resilient organisations assume failures will occur and design systems that contain them quickly.

This philosophy reflects another engineering principle, ALARP, or as low as reasonably practicable. Rather than pursuing the impossible objective of eliminating all risk, organisations should reduce risk through practical, proportionate, and measurable controls. Governance is not about achieving perfection. It is about ensuring failures remain manageable.

Perhaps the most significant insight from the discussion is that successful AI governance is ultimately cultural, rather than technical.

High-reliability organisations continually discuss operational risk. They investigate weak signals before they become major incidents. Monitoring is continuous, learning never stops, and accountability is clearly understood. Safety is not delegated to a policy document. It becomes part of everyday operations.

AI demands the same mindset.

As enterprises move beyond pilots and proofs of concept, governance must evolve from compliance to capability. The organisations that create lasting value from AI will not necessarily have the largest models or the most sophisticated algorithms. They will be the organisations that build resilient operational systems around those technologies.

The future of AI governance will not be written solely in policy manuals.

It will be measured by how effectively organisations monitor, adapt, intervene, and continuously improve the intelligent systems that increasingly underpin modern business.

About the Contributors

Jan Esman
Jan Esman is a seasoned digital advisory leader with extensive experience in IT strategy, enterprise architecture, and digital transformation. His expertise in aligning technology solutions with business objectives ensures that CTO Consulting clients benefit from strategic insights and effective digital initiatives.

Dale Rogers
Dale Rogers is a Service Design Leader specialising in digital transformation across utilities, transport, and government. With expertise in strategic planning, agile delivery, and human-centred design, Dale translates complex systems into actionable change with a keen eye on emerging technologies like AI, blockchain, and IoT.