A Phased Governance Rollout for AI Agents: Crawl, Walk, Run

By Kehinde Ogunlowo · July 27, 2026

A Phased Governance Rollout for AI Agents: Crawl, Walk, Run

There are two tracks to getting AI agents into an enterprise. The engineering track — orchestration, RAG, guardrails, evaluation — is the one most teams plan for, and we covered it in detail in AI Agents in Enterprise: From POC to Production. The governance track runs alongside it and answers a different question: not "does the agent work?" but "how much are we willing to let it do, and what evidence justifies each increase?"

This post is about the governance track. The core idea is simple: autonomy is earned with evidence, phase by phase, against criteria written down before the phase begins. Organizations that skip this either stall — pilots that never touch anything real — or lurch, granting write access on enthusiasm and revoking it after the first incident. A phased rollout is how you avoid both.

Why Phases, Not a Single Approval

A single go/no-go decision for an agent forces the risk owner to approve everything the agent might ever do, based on a demo. That is an impossible ask, so risk owners rationally say no, and programs die in review.

Phases change the shape of the decision. Each phase grants a bounded set of capabilities, runs long enough to produce evidence, and ends with a graduation decision against pre-written criteria. The risk owner approves one increment at a time, with data. This is the operating pattern behind governed AI agents, and it maps cleanly onto the risk-management framing regulators and standards bodies use: the NIST AI RMF's measure-and-manage loop, ISO/IEC 42001's continual improvement cycle, and — at a general level — the EU AI Act's expectation of human oversight proportionate to risk. None of those mandate this exact three-phase shape; all of them are easier to evidence if you have it.

Phase 1 — Crawl: The Read-Only Pilot

What the agent may do: retrieve, summarize, answer, draft. Nothing that changes state anywhere. Read-only credentials enforced at the identity layer, not by prompt instructions — the agent physically cannot write, regardless of what it decides to attempt.

Controls in place from day one:

• Full session logging. Every model call, retrieval, and tool invocation recorded with the initiating user. Building the audit trail in phase 1, when stakes are lowest, means it is proven by the time it matters. (Our post on audit trails for AI agents covers the ledger design.)
• A bounded user group. Named participants who know they are using a pilot, with a channel for flagging bad outputs. Not a quiet company-wide rollout.
• Human review sampling. A defined fraction of sessions reviewed by domain experts each week, with findings logged — this becomes your baseline quality evidence.
• A registered system. The agent goes into your AI system register with an owner, a purpose, and a data classification, even though it is "just a pilot." Unregistered pilots are how shadow AI starts.

Graduation criteria — written before the pilot starts. For example: a defined minimum accuracy on the human-reviewed sample, sustained over a set number of weeks; zero incidents of the agent exposing data the requesting user could not already access; the audit trail successfully answering a reconstruction exercise ("show us everything session X did"); and user feedback indicating the agent is actually useful, because governing a useless agent is wasted effort.

The point of writing criteria first is that it converts graduation from a mood into a checklist, and it commits the organization in both directions — the agent that meets criteria advances, and the agent that does not, does not, regardless of sponsorship.

Phase 2 — Walk: Supervised Writes

What the agent may do: a short allowlist of state-changing actions — create a ticket, draft-and-queue an email, update a specific field — each individually approved into scope. Not "write access" as a category; specific verbs on specific objects.

Controls added:

• Human approval gates. Every write is proposed by the agent and confirmed by a person who sees exactly what will change. The approval itself is logged: who, what they saw, when.
• Reversible actions first. Sequence the allowlist so early write permissions are cheaply undoable (a mislabeled ticket) and defer expensive ones (customer-facing sends, financial records) to late in the phase or to phase 3 review.
• Rate and blast-radius limits. Ceilings on writes per session and per hour, enforced outside the agent. A misbehaving agent should hit a limit, not a headline.
• A tested kill switch. Phase 2 is when the stop mechanism must exist and be drilled — pausing intake, freezing writes, revoking credentials — because phase 2 is the first time there is something worth stopping. The security controls around agent credentials and tool permissions are covered in our AI agent security overview.

Graduation criteria. For example: a defined approval rate — if humans are approving nearly everything the agent proposes over a sustained period, supervision is confirming rather than catching; a bounded rework rate on approved actions; zero guardrail-bypass incidents; kill-switch drill completed with time-to-stop measured; and incident runbooks exercised at least once, even against a synthetic incident.

The approval-rate criterion deserves emphasis because it is the honest signal. If reviewers reject or correct a meaningful share of proposals, the agent is not ready, no matter how good the demo was. If they approve essentially everything for months, you are paying people to click yes — evidence that supervision can be relaxed.

Phase 3 — Run: Bounded Autonomy

What the agent may do: execute its allowlisted actions without pre-approval, inside the same rate limits and permission boundaries, with oversight shifted from gate to monitor.

Controls that replace the approval gate:

• Post-hoc sampling review. A defined fraction of autonomous actions reviewed after the fact, with the fraction increasing automatically if quality dips.
• Anomaly alerting. Volume spikes, unusual action patterns, guardrail-trip-rate increases page a human. The agent operates alone; it is never unwatched.
• Periodic recertification. On a fixed cycle — and on every material change to the model, prompt, or toolset — the agent's autonomy is re-confirmed against current evidence. Autonomy granted once and never revisited is how last year's safe agent becomes this year's finding.
• Irreversibility ceiling. Some actions never graduate to autonomy: irreversible, high-value, or externally binding operations keep a human gate permanently. Deciding which actions sit above the ceiling is a risk-owner decision, made explicitly, not a default reached by drift.

Phase 3 is not "full autonomy." It is supervised autonomy with the supervision moved to where it scales. The enterprise deployments that hold up under audit are the ones that can show the monitor, the sampling records, and the recertification history — not just the original approval.

Graduation Criteria Are Governance Artifacts

Three properties make phase criteria real rather than decorative:

1. Written before the phase begins. Criteria drafted at graduation time will be drafted to fit whatever happened.
2. Signed by the risk owner, not the build team. The people building the agent propose; someone accountable for the risk — security, a governance function, the business owner of the affected process — disposes. Your broader AI governance structure should name this role explicitly.
3. Evidenced from the ledger, not the demo. Every criterion should be checkable from logged data: review samples, approval records, incident tickets, drill timings. If a criterion cannot be evidenced from records, rewrite it until it can.

Anti-Patterns

Watch for these; each one has stalled or burned real programs:

• Pilot purgatory. The agent stays read-only for a year because no one wrote graduation criteria, so no one can say yes. The absence of criteria does not reduce risk — it just converts governance into an indefinite no.
• Big-bang autonomy. Skipping phase 2 because approvals feel slow. The supervised-writes phase is where you learn the agent's real failure modes at survivable cost; teams that skip it learn them in production.
• Paper governance. A policy document with no enforcement point. If "the agent must not write to finance systems" is enforced by a sentence in the system prompt rather than by credentials, it is a wish, not a control.
• Shared service accounts. Actions that cannot be attributed to a specific agent and initiating user make every later phase unauditable. Identity is a phase-1 requirement, not a hardening task.
• Governance owned by the build team. Builders grading their own graduation reliably graduate. Separation of duties is not bureaucracy here; it is what makes the evidence credible.
• Criteria inflation after failure. An agent misses graduation, and the response is to relax the criteria rather than fix the agent. Once criteria bend, every future graduation is negotiable.

Where to Go From Here

The crawl-walk-run structure is the rollout half of a governance program; the other half is the standing control set each phase draws on — agent identity, scoped permissions, session ledgers, oversight roles, and kill-switch design. Both halves are documented in the Enterprise AI Agent Blueprint, a free guide written to be handed directly to your security and risk teams as the starting checklist.

If you have an agent stuck in pilot purgatory, or a rollout plan that needs graduation criteria a risk owner will actually sign, book a call and we will work through the phases against your specific deployment.