Why governance comes first

Building the demo is quick. You connect a chat model to a search index, write a prompt, and after an afternoon you have something that answers questions from the internal knowledge base. Then someone asks whether it can go to production.

Once an assistant reads internal data, or an agent is allowed to change something, the open questions are mostly not about the model. Where is the data processed? Who may ask what? What happens on a jailbreak attempt? Can you show, six months later, who approved a given action?

This year I built an internal AI platform on Microsoft Foundry. The model was maybe 10% of the work and governance was the rest. This post describes the base architecture. Later posts in the series cover the guardrails, the agents and the approval cockpit.

I had already set up my own AI tooling as infrastructure (described here). Doing it for a company is different, because the parts you can skip for yourself are the ones an auditor asks about.

I build Azure landing zones for a living, and this is the same idea applied to AI: teams build on a base that already handles security, networking and compliance, so they don’t have to solve those again per project.

One foundation, two use cases

I decided early not to build two separate platforms. There is one hardened foundation with two quite different applications on top.

  • Employee assistant. Self-service for everyone in the company: RAG answers over the internal knowledge base, a voice interface, SSO and guardrails on every message. A single request can’t do much damage, but it touches sensitive content and many users.
  • Engineering agents. Agents that do engineering work: reports on module versions and drift, impact analysis, and writing infrastructure code. Because they can propose changes, they get the tightest restrictions. The propose-only post covers how.

Both use the same gateway, guardrail policy, identity model and audit trail.

One foundation, two faces Two audiences and two risk profiles on one governed base FACE 1 Employee Assistant RAG over the knowledge base Realtime voice · Entra SSO FACE 2 Engineering Agents Reporting · blast-radius analysis Terraform authoring (propose-only) Governed AI Gateway (APIM) fronts every model and tool call · tiers access · audits HARDENED FOUNDATION Microsoft Foundry models · RAG · agents Guardrails content filter + Azure Policy Identity Managed Identity + Entra RBAC Key Vault the few unavoidable secrets Observability Log Analytics + App Insights
Both experiences route through one governed gateway down to a shared, hardened foundation.

What the foundation contains

None of the components are unusual. What matters is that every capability goes through them, so governance is part of the platform and doesn’t have to be added per project.

Piece What it does Why it stays in the base
Microsoft Foundry account Hosts the chat, embedding and agent models, plus RAG and the agent runtime The models run where the account is, so it determines data residency
AI Search The RAG index over the knowledge base Sensitive content flows through retrieval, so it belongs inside the governed boundary
API Management (AI gateway) Fronts all model and tool traffic One place to tier access, enforce allow-lists and log every call
Custom content filter + Azure Policy Harm, jailbreak and indirect-attack guardrails, enforced org-wide Projects can’t switch it off
Entra groups + project RBAC Per-identity access to models, tools and projects Access is granted by identity, not by network location
Key Vault Holds the few secrets that can’t be avoided Some secrets remain; they should be few and in one place
Log Analytics + Application Insights Central audit and observability Only structured events can be queried later

The rule is that every new tool, model deployment or agent routes through the gateway, inherits the guardrail policy and writes an audit event. If a feature only works by bypassing the base, it doesn’t ship. Without that rule the platform slowly collects ungoverned side channels.

Data residency is decided by the region

This is often misunderstood. In Azure AI, the region of the model deployment is the residency boundary: inference happens where the account is. The gateway in front controls access and logging. It doesn’t move data, and it doesn’t make a model in another region compliant.

If your requirements say customer data stays in the country, you pick an in-country region for chat models, embeddings and the RAG index. That works until you need a feature the region doesn’t offer yet.

For me that was realtime voice. The in-country region had the chat, embedding and agent models, but not the realtime voice model. Running everything in a region that has voice would have been simpler. Instead I split it: the data-bearing workloads stay in-country, and only the voice model runs in an EU DataZone region, where the data zone keeps that traffic inside the EU. That gives two accounts on one platform and a residency setup that holds up in an audit.

Residency is a region decision The model's location sets the residency boundary AI Gateway governs access and audit, never residency In-country region · e.g. Switzerland North Chat, RAG & agents Chat model (GPT) Embedding model AI Search, RAG index Engineering agents Customer data and the RAG index stay in-country. EU DataZone region · e.g. Sweden Central Realtime voice only Realtime voice model Why split at all? The voice model isn't offered in-country. An EU DataZone deployment keeps that traffic inside the EU instead of leaking the whole workload to a far region. Same guardrail policy, copied per account.
Split the workload by region so the data-bearing parts stay in-country and only voice runs in the EU DataZone.

One detail caught me out: the guardrail policy is scoped to the account. The voice account in the second region needs its own copy of the content filter before the voice deployment can reference it. You only notice when a deployment fails.

Managed identities instead of secrets

A static key in an app setting is the quickest way to undermine the whole setup. So every component uses a managed identity, and Key Vault holds only the few secrets that can’t be replaced by one.

The assistant, the agents, the gateway and the background jobs each authenticate with their own managed identity, with RBAC scoped to what they need. There are no shared service principals with client secrets and no API keys copied between services. For the people operating these components there is no standing privilege either; the platform relies on a PIM setup for that. When one component calls another on a user’s behalf, it uses an on-behalf-of token exchange, so the user’s identity is passed along instead of a machine identity. That way every action can be traced to a person, which the approval cockpit post builds on.

Everything fails closed. A missing config value or identity claim means no access, never a fallback to something more permissive. The hard part is turning down the small exceptions that would make a demo easier.

Central audit from the start

Every guardrail decision, every tool call through the gateway and every approval is written to Log Analytics and Application Insights as a structured event: who, what, which resource, when. Plain log.info lines are not enough, because you can’t query them properly later.

My check: in six months, can I answer “show me every action this agent took against that repository last quarter, and who approved each one” with one query? Building that in at the start is much cheaper than reconstructing it from application logs when someone asks during a review.

This is the same argument as for an Azure cloud foundation. The groundwork decides whether the application survives production. With AI the consequences just show up sooner.

Downsides

Governance-first is the right approach, but it has costs.

  • Control-plane serialization. Azure serializes control-plane operations on a single Foundry account. Creating the guardrail, deployments and projects in parallel returns 409 RequestConflict, so you chain them explicitly. With infrastructure as code this is easy to run into.
  • Region availability. The voice split works, but a model missing in your preferred region keeps coming up. For every new model you check the region matrix first.
  • Gateway tier and cost. A basic APIM tier is fine for a proof of concept. Production needs the throughput and resilience to sit in the path of every call, and that costs real money.
  • Overhead comes before the benefit. Guardrails, allow-lists and audit wiring add nothing to the first demo. They pay off later, usually at the first incident.

For anything that touches real company data or infrastructure, I think it is worth it. For a throwaway prototype it isn’t.

Next posts in the series

  • Guardrails. The custom content filter (harm thresholds, jailbreak and indirect-attack shields) and the governed MCP gateway that tiers tool access and issues short-lived tokens per call.
  • Propose-only agents. How an agent can write Terraform without being able to merge, with the human review as the only merge path.
  • Human-in-the-loop approvals. The approval cockpit: a feed per person, explicit confirmation for approvals, and an audit event for every decision.

If you are working on something similar or disagree with one of these decisions, let me know.

Resources

  1. Microsoft, “Microsoft Foundry documentation,” Microsoft Learn
  2. Microsoft, “Data, privacy, and security for Foundry Models sold by Azure in Microsoft Foundry,” Microsoft Learn
  3. Microsoft, “Content filtering for Microsoft Foundry Models,” Microsoft Learn
  4. Microsoft, “GenAI gateway capabilities in Azure API Management,” Microsoft Learn
  5. Microsoft, “Managed identities for Azure resources,” Microsoft Learn