Everyone wants the assistant. Nobody wants to own the audit log.
The demo is always the easy part. You wire a chat model to a search index, drop in a nice prompt, and within an afternoon you have something that answers questions from your company’s knowledge base. People are impressed. Somebody asks “can we ship this?”
That is where it gets real. The moment an AI assistant touches internal data, or an agent gets permission to change something, the hard questions have nothing to do with the model. Where does the data physically live? Who is allowed to ask what? What happens when someone tries to jailbreak it? Can you prove, six months later, who approved which action?
I spent a good chunk of this year building an internal AI platform on Microsoft Foundry, and the biggest lesson is embarrassingly boring. The model is maybe 10% of the work. The other 90% is governance. This post is the architecture I landed on. Later posts in the series go deep on the guardrails, the agents, and the human-in-the-loop cockpit, but you need the foundation first.
I had already been treating my own AI setup as infrastructure rather than a chat window — that experiment is where the instinct came from. Doing the same for a company is a different problem, because the parts you can skip for yourself are exactly the parts an auditor asks about.
I build Azure landing zones for a living, so here is the analogy I can’t help using. This is a landing zone, but for AI. The whole point of a landing zone is that teams can build on top of it without re-litigating security, networking, and compliance every time. A governed AI platform does the same job for AI workloads.
One foundation, two faces
The core design decision came early. Don’t build two AI platforms. Build one hardened foundation, and let two very different experiences sit on top of it.
- Face 1, the employee assistant. A self-service assistant for everyone in the company: retrieval-augmented answers over the internal knowledge base, a voice interface, single sign-on, and guardrails on every message. The blast radius per request is low, but it touches sensitive internal content and a lot of users.
- Face 2, the engineering agents. Governed agents that do real engineering work: reporting on module currency and drift, blast-radius analysis, and authoring infrastructure code. The blast radius here is high, because these can propose changes, so they get the tightest leash. I cover exactly how in the propose-only post.
Two audiences with very different risk profiles, sitting on one governed base. Both go through the same gateway, the same guardrail policy, the same identity model, the same audit trail. That is the diagram I kept coming back to.
If you take one thing from this post, take that shape. Everything else is detail hanging off it.
The foundation, piece by piece
None of the individual components are exotic. The discipline is that every capability runs through them, so governance is a property of the platform rather than something you bolt on per project. Here is what sits in the base.
| Piece | What it does | Why it stays in the base |
|---|---|---|
| Microsoft Foundry account | Hosts the chat, embedding, and agent models, plus RAG and the agent runtime | The models live where this account lives, which makes it your data-residency anchor |
| AI Search | The RAG index over the knowledge base | Retrieval is where sensitive content actually flows, so it belongs inside the governed boundary |
| API Management (AI gateway) | Fronts all model and tool traffic | One choke point to tier access, enforce allow-lists, and record every call |
| Custom content filter + Azure Policy | Harm, jailbreak, and indirect-attack guardrails, enforced org-wide | A guardrail you can switch off per project stops being a guardrail |
| Entra groups + project RBAC | Per-identity access to models, tools, and projects | Access is an identity question, never a network shortcut |
| Key Vault | Holds the handful of secrets that can’t be avoided | Keep secrets to a minimum instead of pretending you have none |
| Log Analytics + Application Insights | Central audit and observability | If it isn’t logged as a structured event, you can’t answer questions about it later |
The mental model I use is governance before feature. Every new capability, whether that is a tool, a model deployment, or an agent, has to route through the gateway, inherit the guardrail policy, and emit an audit event. If a feature only works by going around that base, it doesn’t ship. That one rule keeps the platform from quietly rotting into a pile of ungoverned side-channels.
Data residency is a region decision, not a gateway decision
People get this one wrong, so it gets its own diagram. In Azure AI, the model deployment’s region is the residency boundary. Where the account lives is where the inference happens and where the data goes. The gateway in front of it governs access and audit. It does not move the data anywhere, and it does not make a model in another region suddenly compliant.
That matters the second your requirements say customer data stays in the country. You pick an in-country region for the chat models, the embeddings, and the RAG index, and you are done, right up until you want a feature that region doesn’t offer yet.
For me that feature was realtime voice. The in-country region carried the full chat, embedding, and agent model set, but not the realtime voice model. The lazy fix is to shrug and run everything in whatever region has voice. The better fix is to split. Keep the data-bearing workloads in-country, and put only the voice model in an EU DataZone region, where the data-zone guarantee keeps that traffic inside the EU. Two accounts, one platform, and a residency story you can actually defend in an audit.
One subtlety that bit me: the guardrail policy is account-scoped. So the voice account in the second region needs its own copy of the same content filter before the voice deployment can bind to it. Same rules, two places. It is the kind of detail that never shows up in a diagram until you forget it and a deployment fails.
Secret-less by default
The fastest way to turn a governed platform into an ungoverned one is a static key sitting in an app setting. So the rule here is Managed Identity everywhere, and Key Vault only for the handful of secrets that genuinely can’t be an identity.
Concretely: the assistant, the agents, the gateway, and the background jobs all authenticate as their own managed identity and get RBAC-scoped to exactly what they need. No shared service principals with a fat client secret. No API keys pasted between services. For the human operators behind those components the equivalent rule is no standing privilege either — that’s the PIM setup this platform assumes underneath it. When one component needs to call another on a user’s behalf, it uses an on-behalf-of token exchange so the user’s identity flows through instead of a machine’s. That feeds straight into the human-in-the-loop story in a later post, because every action ends up attributable to a real person.
The principle underneath it is fail-closed. A missing config value or a missing identity claim means no access, not a quiet fallback to something permissive. That sounds obvious written down. The discipline is refusing every tempting little exception that would make a demo smoother.
Central audit from day one
The last foundational piece is audit, and it is the one that pays off latest and biggest. Every guardrail decision, every tool call through the gateway, and every approval lands in Log Analytics and Application Insights as a structured event: who, what, which resource, and when. Not a stray log.info line you will never be able to query.
Here is the test I hold it to. Six months from now, can I answer “show me every action this agent took against that repository last quarter, and who approved each one” with a single query? If the answer is no, the audit is just decoration. Building it in from day one is far cheaper than reconstructing it from application logs after someone asks the question in a review.
None of this is new thinking, incidentally. It’s the same argument as the Azure cloud foundation: the unglamorous groundwork decides whether the interesting part survives contact with production. AI just makes the bill arrive faster.
What this actually costs you
I would be a poor sparring partner if I only sold you the upside. Governance-first is the right call, but it is not free.
- Control-plane serialization. Azure serializes control-plane operations on a single Foundry account, so creating the guardrail, the deployments, and the projects in parallel gets you
409 RequestConflict. You end up chaining them explicitly. It is annoying, and easy to trip over with infrastructure-as-code until you learn it. - Region gymnastics. The voice split is clean once it exists, but “one feature isn’t in your preferred region” is a recurring tax. Every new model you want, you check the region matrix first.
- Gateway tier and cost. A real AI gateway is not the cheapest SKU. For a proof of concept you can run a basic tier. Production wants the throughput and resilience to sit in the hot path of every call, and that is a real line item.
- The overhead is felt before it is appreciated. Guardrails, allow-lists, and audit wiring do nothing for the first demo. They make the hundredth deployment safe. That is a hard sell right up until the first incident, when it is suddenly the only thing anyone cares about.
Is it worth it? For anything touching real company data or real infrastructure, absolutely. For a throwaway prototype, honestly no. Don’t cargo-cult this onto a weekend experiment.
Where this series goes
This was the foundation. The interesting, opinionated parts build on top of it, and each gets its own post.
- Guardrails that actually hold. The custom content filter (harm thresholds, jailbreak and indirect-attack shields), and the governed MCP gateway that tiers tool access and mints short-lived tokens per call.
- Propose-only agents. How you let an agent write Terraform without ever giving it the right to merge, and why the human stays the single, enforced merge path.
- Human-in-the-loop by design. The approval cockpit: a person-centric feed, hard-confirm approvals, and an audit event behind every decision.
The thread running through all of it is the same one from the top of this post. The model is the easy 10%. If you are building anything past a demo, budget for the other 90%, and build the foundation first.
If you are working through similar decisions, or you think I got one of these calls wrong, I’d genuinely like to hear it.
Resources
- Microsoft, “Microsoft Foundry documentation,” Microsoft Learn
- Microsoft, “Data, privacy, and security for Foundry Models sold by Azure in Microsoft Foundry,” Microsoft Learn
- Microsoft, “Content filtering for Microsoft Foundry Models,” Microsoft Learn
- Microsoft, “GenAI gateway capabilities in Azure API Management,” Microsoft Learn
- Microsoft, “Managed identities for Azure resources,” Microsoft Learn