Where Does the Model Run? Deployment Topologies for Enterprise AI

By Kehinde Ogunlowo · July 30, 2026

Quick answer: There are four practical topologies for running AI in an enterprise — vendor SaaS, gateway-mediated SaaS, in-tenancy managed service, and self-hosted in-VPC. They differ mainly in where the prompt and the retrieved context physically travel, and that single fact determines how the security review goes. Choose the topology before you choose the model, because the topology is expensive to change and the model is not.

Last updated: July 2026 | Author: Kehinde Ogunlowo, Principal AI Platform Architect

Most AI pilots do not fail on model quality. They fail in the week the architecture review board asks where the customer data goes, and nobody has a diagram that answers it.

The question sounds like a compliance formality. It is not. It decides your cost model, your latency floor, your vendor lock-in, your incident-response story, and whether the deal closes this quarter or next year. And it is the one decision that is genuinely hard to reverse: swapping Claude for GPT is an afternoon; moving from vendor SaaS to in-tenancy inference is a re-platform.

If your job title is on this list, this decision is partly yours:

The four topologies

1. Vendor SaaS

The application calls a model provider's public API directly. Fastest to build, no infrastructure, and the option every pilot starts on.

What you are accepting: prompts and retrieved context leave your network to a third party, egress is to a public endpoint, and your data-processing agreement is whatever the provider publishes. For a regulated workload this is usually where the review stops.

Reasonable when: the data is public or synthetic, you are proving value, or the buyer has no regulatory surface.

2. Gateway-mediated SaaS

Same provider, but every call passes through a gateway you run — an egress proxy that authenticates the caller, enforces per-team quota, redacts what should not leave, and logs the full request and response.

This is the most under-used topology and the best value for effort. You keep SaaS economics and add the three things a security review actually asks for: an audit trail, a policy enforcement point, and a kill switch. It is also the only topology that makes switching providers cheap later, because your application talks to the gateway, not the vendor.

What it costs: one more hop of latency, and a service you now have to run and keep available.

3. In-tenancy managed service

The model runs inside your own cloud account, on a managed service — Amazon Bedrock, Azure OpenAI, or Google Vertex AI. Traffic reaches it over a private endpoint. It never crosses the public internet.

This is where most regulated enterprises land, and for good reason: the data stays inside a boundary the customer already trusts and already audits, keys can be customer-managed, and the provider's compliance attestations are inherited rather than re-argued. You give up access to the newest model the day it launches, and you accept the provider's regional availability.

4. Self-hosted in-VPC

Open-weight models on your own compute, typically vLLM or Ray Serve on Kubernetes with GPU node pools.

Total control, and the only option when a model must be fine-tuned on data that cannot leave, or when an air-gapped environment is a hard requirement. It is also the most expensive by a wide margin, and the cost is not the GPUs — it is the people. You have now signed up to own model serving, GPU scheduling, capacity planning, and an upgrade path, permanently.

The comparison that matters

The flow the review board is actually asking about

The question is never "which model." It is "trace one request." Be able to draw this without hesitating:

User
  |
  v
Application  ---- authenticates the human
  |
  v
Gateway ----------- policy: who may call, at what rate, with what data
  |                 redaction, logging, kill switch
  v
Retrieval --------- the corpus, and who is allowed to see each document
  |
  v
Model endpoint ---- private or public? whose account? which region?
  |
  v
Tool / action ----- can it WRITE anywhere? with whose credentials?
  |
  v
Audit log --------- attributable to one identity, immutable, retained

Two boxes decide the outcome. Model endpoint is where the data-residency conversation happens. Tool / action is where the security conversation happens, because a system that can only read is a very different risk from one that can write to a ticketing system, a repository, or a payment ledger.

Choosing, in the order that saves rework

Ask these in sequence and stop at the first answer that binds.

1. Can the data legally leave the tenancy? If no, you are choosing between topologies 3 and 4. Nothing else matters yet.
2. Must the model be fine-tuned on data that cannot leave? If yes, topology 4.
3. Does an auditor need a per-request log tied to a named identity? If yes, you need a gateway regardless of the other three answers.
4. Is the workload latency-sensitive at the tens-of-milliseconds level? If yes, self-hosting near the data may be the only option.
5. Otherwise: gateway-mediated SaaS to start, with the gateway abstraction in place so moving to in-tenancy later is a configuration change rather than a rewrite.

That last point is the one worth internalising. The gateway is cheap insurance. Teams that skip it because "we will move to Bedrock eventually" end up with the vendor's SDK woven through forty files.

What this looks like in Terraform

Citadel publishes the modules for the in-tenancy topology on each major cloud. They are open source, and reading them is faster than reading the provider documentation:

• terraform-aws-bedrock-platform — Bedrock with knowledge bases, agents, guardrails, and OpenSearch Serverless
• terraform-azure-openai-platform — Azure OpenAI with model deployments, private endpoint, and APIM in front
• terraform-gcp-vertex-ai-agents — Vertex AI Agents with Dialogflow CX
• multi-cloud-landing-zone — the identity federation and network layer underneath all three

A workable repository layout for the platform itself:

ai-platform/
├── gateway/              # the policy enforcement point
│   ├── policy/           # who may call, quota, redaction rules
│   ├── routing/          # provider abstraction — the switching cost
│   └── audit/            # per-request, attributable, append-only
├── endpoints/
│   ├── bedrock/          # private endpoint + CMK
│   ├── azure-openai/
│   └── self-hosted/      # vLLM on GPU node pools
├── retrieval/
│   ├── corpus/           # with document-level ACLs
│   └── index/
├── identity/             # workload identity per calling service
├── network/              # egress allow-list, default deny
└── evidence/             # the artifacts the review board asks for

The evidence/ directory is not decoration. If you cannot produce a diagram, an egress allow-list, and a sample audit record on request, the review stalls regardless of how good the architecture is.

Where teams get this wrong

Choosing the model first. The model is a swappable dependency. The topology is an architectural commitment. Deciding in that order guarantees rework.

Treating the pilot topology as the production topology. Vendor SaaS is the correct choice for a two-week proof of value and the wrong one for a regulated production workload. Plan the migration at the start, not when procurement asks.

Forgetting the tool surface. Teams spend weeks on where the model runs and minutes on what it is allowed to write to. A read-only assistant on vendor SaaS is often a lower risk than a write-capable agent running in-tenancy. Model location is one axis; blast radius is the other, and reviewers care more about the second.

No gateway. Retrofitting a policy enforcement point after the application has hardcoded a provider SDK is the most common avoidable rewrite in this space.

Frequently asked questions

Is in-tenancy always more secure than SaaS? No. It changes who holds the risk rather than removing it. In-tenancy means the data stays in a boundary you control, but you have also taken on responsibility for the configuration of that boundary. A misconfigured private endpoint with an over-permissive IAM role is worse than a well-governed SaaS call through a logging gateway. The advantage of in-tenancy is that it is auditable by the customer's existing controls, which is usually what the review is really asking for.

Do we need a gateway if we are already on Bedrock or Azure OpenAI? Usually yes, for a different reason. In-tenancy solves data residency; it does not by itself give you per-team quota, prompt redaction, cross-provider routing, or a single place to revoke access. Those are gateway concerns and they remain useful no matter where the endpoint lives.

What does customer-managed key (BYOK) actually buy? Cryptographic control of data at rest and, more practically, the ability to revoke. It matters most when the contractual requirement is that the customer can render data unreadable independently of the provider. It is not a substitute for network controls or access policy, and it is often requested as a proxy for "we want a kill switch" — worth clarifying which one the reviewer means.

Which topology should a proof of concept use? Gateway-mediated SaaS. It is nearly as fast to stand up as raw vendor SaaS, and it means the production migration is a routing change rather than a rewrite. The gateway is the cheapest architectural decision on this page.

How do we handle multiple clouds? Standardise the gateway, not the model. Provider APIs diverge and will keep diverging; a routing layer that speaks one internal contract is what makes a multi-cloud AI estate maintainable. The multi-cloud landing zone handles the identity federation underneath.

Where to go next

The topology decision sits upstream of the security review. Once it is made, the review becomes a question of evidence rather than architecture — how to get AI agents through enterprise security review covers what that evidence looks like, and autonomy tiers for AI agents covers the write-surface question that reviewers care about most.

If the identity layer is the open question, workload identity for AI agents covers SPIFFE, token exchange, and why service accounts do not survive contact with agents.

For the role-based reading paths across architecture, platform, security and governance, see the Citadel knowledge hub. The AI Architect track and the AI Platform Engineering track both start here.

If you would rather not work this out from first principles on a live programme, the Agent Assurance Assessment is three weeks at a published price and ends with a register, an evidence pack, and one governed agent actually running. Or book a discovery call and we will scope it.

References

• AWS Well-Architected Machine Learning Lens
• Azure Architecture Center
• Google Cloud Architecture Framework
• NIST AI Risk Management Framework
• NIST SP 800-207, Zero Trust Architecture
• OWASP Top 10 for LLM Applications