Across many of the tenants I work with, the rebranding to Azure AI Foundry at Ignite 2024 has given AI initiatives a clear home in Azure. The old governance problems haven't gone away, though. If anything, they've become more complicated. Teams still have to decide who may deploy their own models, how to avoid a sprawl of API keys and untracked access, and how to keep costs from running away without blocking experimentation. Azure AI Foundry governance is what makes responsible AI possible, and in practice you need it before you can scale innovation safely.

Hubs, projects and roles define who can access what

The new hub/project architecture in Azure AI Foundry (overview) finally brings some order to what early OpenAI rollouts looked like. A Foundry hub is a container for AI projects. It lets you define network, policy, budget and access boundaries centrally for a team or business unit. RBAC is much more granular than before. Azure AI Developer, Azure AI User/Inference User and Cognitive Services OpenAI User each grant a different level of access, from full model deployment down to query-only use (RBAC in detail). Wherever roles aren't assigned deliberately, I see a mess. Give too many users broad access to training data or model definitions and you will run into trouble sooner or later. As with any other Azure resource, you can automate role assignments at hub and project level through groups, which makes onboarding much easier.

Managed identities should replace API keys for authentication

If your developers are still pasting API keys into their notebooks, it's time to change that. Azure AI Foundry lets you turn off local authentication (API key access) with the disableLocalAuth setting on each Cognitive Services instance. After that, only managed identities or service principals can reach the models, which cuts the risk of leaked keys or accidental misuse considerably. For apps running in Azure that call OpenAI models, managed identities are now best practice, much as I described in my deep-dive on Entra ID workload identities. The settings are in the Azure portal. In your Cognitive Services instance, disable local authentication under "API keys", then assign a managed identity under "Identity" and grant access through RBAC (How-to).

Pro tip: Always enable disableLocalAuth and give every workload its own managed identity. Access stays auditable, and incident response gets a lot simpler.

Private networking, Azure Policy and AI filters close the remaining gaps

For production, I strongly recommend turning on private endpoints and managed virtual networks for every Foundry hub. That keeps internal systems and sensitive training data off the public internet. Surprisingly few organizations have fully done this so far. Azure Policy is also very useful here. There are dozens of built-in policies for Cognitive Services and AI Foundry that can restrict regions, require network isolation or make disabled local auth the default. Assign them at management group or subscription level and governance is in place from day one.

At the heart of Azure AI Foundry governance sit the configurable content filters and Prompt Shields. They block unwanted prompts, outputs or content in real time, before internal data or hallucinated answers end up outside the organization. You can tune the filters per hub and project, by language, type or other criteria.

This simple diagram shows where the governance controls apply when someone accesses a model:

flowchart TD
User["User with role"] --> App["App with Managed Identity"]
App --> Hub["Foundry Hub"]
Hub --> VNet["Private VNet/Endpoint"]
Hub --> Policy["Azure Policy"]
Hub --> Filter["Content Filter / Prompt Shield"]
Filter --> Model["AI Model deployment"]

Figure: Governance controls applied when a model in Azure AI Foundry is accessed.

Quotas, budgets and deployment approvals keep costs under control

Cost is the other problem that keeps coming up. Without proper controls, AI spending can climb very quickly. Azure AI Foundry lets you set quotas and budgets per hub and project. The teams I've seen do this best enforce quotas on API usage, number of models or total compute hours, which stops "runaway" jobs and unplanned overages early. Larger organizations should also connect Azure Budgets and Cost Alerts so they can see what's happening. Each hub also supports workflow-based approval for new model deployments. A model only goes live once a designated reviewer has signed off, which matters for compliance and prevents accidental releases.

If you take enterprise security for GPT-style models seriously, have a look at my post on securing and governing Azure OpenAI. A lot of what I wrote there applies directly to Foundry hubs: audit prompt logs, enforce data classification and integrate with your SIEM wherever you can.

Good governance speeds up AI projects in my experience

One thing stands out from my recent projects. The better Azure AI Foundry governance is defined, the faster and safer AI projects get going. Investing early in clear roles, network controls, budgets and policy enforcement pays off, as long as everyone involved understands how the new mechanisms work. Governance protects you from risk, but when it gives people freedom within sensible controls, it also moves responsible AI work forward.