When I help customers roll out Azure OpenAI, the question of how to secure it stops being theoretical very quickly and turns into something painfully practical. Microsoft 365 Copilot has moved generative AI into the middle of day-to-day business. Even so, I keep running into surprisingly open configurations, including in large environments. API key authentication is still switched on, public network access is enabled, and hardly anyone is watching for misuse. A properly secured Azure OpenAI setup goes well past the defaults and pays close attention to identity, network, data and cost controls.

Entra ID should replace API keys for authentication

I still find plenty of Azure OpenAI deployments with local API key authentication enabled. That is a serious risk, because API keys are hard to control and leak easily. Microsoft has recommended for a long time that you disable local authentication and move to Microsoft Entra ID access control, using managed identities and the Cognitive Services OpenAI User role. Every app or user identity then has to be explicitly authorized, every access can be audited, and you no longer have API keys scattered around. Microsoft documents the details in its guide to using managed identities.

Practical tip: Once you have switched to Entra ID auth, test thoroughly with different managed identities and user accounts. That way you catch missing role assignments or misconfigured apps before go-live.

The OpenAI User role is assigned in the IAM pane

In the Azure portal → Resource group → Azure OpenAI resource → Access control (IAM), assign the Cognitive Services OpenAI User role to the managed identities or users that need it. Disable "local authentication" in the same pane. For any new deployment I now treat this as mandatory.

Private endpoints and correct DNS cut off public access

Network security is not something you can skip. If you want Azure OpenAI locked down, you need private endpoints, which limit API access to your internal Azure virtual networks, and Public network access should always be set to "Disabled". The pitfall I see most often is DNS. If the private DNS zones aren't set up correctly, traffic meant for the private endpoint can still be routed the wrong way. I've covered the usual errors in Private endpoints DNS in Azure: Common mistakes in practice.

This simple diagram shows how a connection secured with a managed identity works together with a private endpoint:

flowchart TD
  App["App with managed identity"] --> Auth["Entra ID authentication"]
  Auth --> MI["Managed identity token"]
  MI --> AI["Azure OpenAI Service"]
  AI --> PE["Private endpoint"]
  PE --> VNet["Virtual network"]

Figure: Authorized Azure OpenAI access using a managed identity and a private endpoint.

One caveat, though. A private endpoint only helps if public network access really is disabled. Many people forget that step and leave a back door open without realizing it.

Content filters and abuse monitoring add guardrails

Security doesn't end at the network boundary. Azure OpenAI comes with built-in content filters that check prompts and completions for problematic content before they are processed. You enable and configure them in your Azure OpenAI resource under "Content Filtering", where you can adjust thresholds and categories such as hate, sexual content, violence and others. The official description of content filtering has the details. In practice I advise reviewing the abuse logs regularly for anything unusual, for example a sudden jump in prompt volume from a particular source.

Data handling and deployment type need to be clear

Data privacy comes up again and again, usually as "Are our prompts used to train Microsoft or OpenAI models?" As of October 2024, prompts and completions from Azure OpenAI customer sessions are not used to train or improve models for Microsoft or OpenAI. That is a real difference from consumer services like ChatGPT.

When you pick model deployments, keep data residency and deployment type in mind. Most models are available regionally or in specific "data zones". Some, particularly newer ones like GPT-4 Turbo, may only be offered as global deployments, which can mean your data is processed outside your preferred region. Before going live, check that the region and model you've chosen actually meet your compliance requirements. Sometimes you will have to accept a compromise.

Azure Policy and quotas enforce standards and control costs

In my experience many organizations make too little use of Azure Policy to enforce baseline standards for OpenAI deployments. Built-in Azure Cognitive Services policies now exist to require private endpoints, restrict allowed regions and more. Assign them at subscription or management group level through the Azure Policy portal. They are an effective way to stop misconfiguration and drift.

Cost control matters just as much. Azure OpenAI is billed pay-as-you-go, but it relies on strict quota limits rather than traditional budgets. You can set quotas in the Azure portal at resource, model or deployment level. They protect you from runaway costs caused by an accidental infinite loop or an unexpected surge in usage.

Securing Azure OpenAI is an ongoing discipline

You can't secure Azure OpenAI by ticking the "Entra ID enabled" box and moving on. If private endpoints, strict role management, ongoing abuse monitoring and hard quotas are missing, you are exposed on the technical, legal and financial side. In the environments I've seen, the weak spots usually sit where identity, network and governance meet. With some rigor, though, Azure OpenAI can be deployed securely and responsibly, and the effort pays for itself as soon as your first real use case reaches production.