If you run multi-agent systems in production, every agent interaction is ultimately API traffic. Every model call, every MCP tool invocation, every agent-to-agent handoff crosses the network as a request that someone needs to authenticate, throttle, log, and pay for. I covered the fundamentals of this earlier — access control, throttling, and observability for LLM endpoints — in my first article on Azure API Management as AI Gateway.

Since Build 2026, Microsoft has extended that story significantly. Azure API Management (APIM) and Azure API Center now reach well beyond language model endpoints: the gateway can import and govern remote MCP servers, A2A agent APIs, and a unified model API that fronts multiple model providers behind a single OpenAI-compatible endpoint. In other words, APIM is evolving from an LLM gateway into the governance plane for the entire agent stack.

This article walks through what actually shipped, what it looks like in code, and where the sharp edges are.

The Problem: Fragmented Governance for Agent Traffic

A production multi-agent system typically fans out across four kinds of endpoints:

  • LLM provider APIs — Azure OpenAI or Microsoft Foundry, Anthropic Claude, Google Vertex AI, each with its own request schema, rate limits, and token accounting
  • MCP tool servers — the databases, SaaS tools, and internal services exposed to agents via the Model Context Protocol
  • A2A agent APIs — agent-to-agent communication, typically JSON-RPC-based, between collaborating agents
  • Traditional internal APIs — the REST services your agents still call underneath everything

In most environments I review, each of these is governed separately — or not governed at all. Rate limits live in application code. Logging is whatever the agent framework emits. Token spend is reconstructed from invoices at month-end. When a runaway agent loop burns through a model quota, or a prompt injection attempt reaches a tool server, you find out after the fact.

The architectural answer is simple to state and hard to build: put one control plane in front of all of it. That is exactly where APIM is heading.

What Actually Shipped

The AI gateway in Azure API Management is not a separate SKU — it is a set of capabilities documented as applying across API Management tiers, with some features limited to the v2 tiers. Here is what matters for agent workloads.

1. Multi-provider model governance

The gateway can manage language model APIs conforming to three schemas:

  • OpenAI Chat Completions and Responses API
  • Anthropic Messages API — currently supported in the APIM v2 tiers
  • Google Vertex AI API

Models can live in Microsoft Foundry or with non-Microsoft providers such as Amazon Bedrock. You import each endpoint once and apply the same policies — throttling, caching, content safety, logging — regardless of provider.

2. Unified model API (preview)

This is the most architecturally significant addition. The unified model API exposes multiple model backends through a single OpenAI-compatible endpoint, handles format translation automatically, and lets you apply governance policies once across all models. Your agents speak one API shape; APIM decides which backend answers, and you can route and fail over across providers without touching application code.

import os
from openai import OpenAI

# One client, one API shape — APIM decides which backend answers
client = OpenAI(
    base_url="https://apim-ai-gw.azure-api.net/llm/v1",
    api_key=os.environ["APIM_SUBSCRIPTION_KEY"],
)

response = client.chat.completions.create(
    model="gpt",  # client-facing alias; APIM maps it to the configured backend
    messages=[{"role": "user", "content": "Draft the incident summary"}],
)
print(response.usage.total_tokens)

Swapping a model, rebalancing between a PTU deployment and pay-as-you-go, or failing over to another provider becomes a gateway configuration change, not a code change. Because it is in preview, I would not build a contractual production dependency on its exact semantics yet — but it is clearly the direction of travel, and worth piloting now.

3. MCP server governance

APIM can now expose existing REST APIs as MCP servers and pass through to existing remote MCP servers. On the inventory side, Azure API Center maintains a registry of remote and local MCP servers — title, description, use cases, runtime URLs — and syncs APIs and MCP servers between APIM and API Center. Registered MCP servers can also be integrated with Microsoft Foundry's tool catalogs, so governed tools become discoverable to Foundry agents.

This matters because MCP sprawl is real. Without a registry, every team wires its agents to whichever MCP servers it happens to know about, and nobody can answer "which tools can our agents reach, and who approved them?"

4. A2A agent API governance

APIM can import A2A agent APIs, bringing agent-to-agent traffic under the same gateway discipline as everything else: authentication, throttling, logging, monitoring. If your architecture has three or more agents collaborating over A2A, this is where you get an audit trail of inter-agent calls instead of a black box.

5. Token controls and semantic caching

Two policy families carry the economics:

  • `llm-token-limit` enforces tokens-per-minute or token quotas per counter key (subscription, IP, or any expression), with optional prompt-token precalculation on the APIM side so over-limit prompts never reach the backend.
  • `llm-semantic-cache-store` / `llm-semantic-cache-lookup` cache completions by vector similarity using the Embeddings API, backed by Azure Managed Redis or any RediSearch-compatible cache. Workloads with repetitive prompts see real reductions in backend token consumption and latency.

6. Token observability and message logging

The gateway emits token metrics with custom dimensions via llm-emit-token-metric, streams prompts and completions to Azure Monitor through diagnostic settings, and tracks token usage per consumer in Application Insights — with a built-in analytics workbook for consumption patterns. That combination is what makes per-agent, per-workflow cost attribution possible.

The Gateway, Deployed

Here is the practical path. First, the infrastructure — pick your weapon:

# Azure CLI: v2 tiers provision through ARM/Bicep, so deploy a template
az group create --name rg-ai-gateway --location malaysiawest

az deployment group create \
  --resource-group rg-ai-gateway \
  --template-file apim-ai-gateway.bicep \
  --parameters apimName=apim-ai-gw

One note before the templates: az apim create has not caught up with the v2 tiers — its --sku-name argument still accepts only the classic SKUs (Developer, Basic, Standard, Premium, Consumption). That is why the CLI path here deploys a template instead of calling the APIM command directly; v2 instances are created through ARM/Bicep, Terraform, or the portal.

# Terraform
resource "azurerm_api_management" "ai_gateway" {
  name                = "apim-ai-gw"
  location            = "malaysiawest"
  resource_group_name = azurerm_resource_group.rg.name
  publisher_name      = "Platform Engineering"
  publisher_email     = "[email protected]"

  sku_name = "PremiumV2_1"
}
// Bicep
resource apim 'Microsoft.ApiManagement/service@2025-09-01-preview' = {
  name: 'apim-ai-gw'
  location: 'malaysiawest'
  sku: {
    name: 'PremiumV2'
    capacity: 1
  }
  properties: {
    publisherEmail: '[email protected]'
    publisherName: 'Platform Engineering'
  }
}

Then the governance layer. A typical inbound policy for an agent-facing model API combines token rate limiting with per-agent metric emission:

<!-- Inbound: cap token throughput per subscription -->
<llm-token-limit counter-key="@(context.Subscription.Id)"
                 tokens-per-minute="20000"
                 estimate-prompt-tokens="false"
                 remaining-tokens-variable-name="remainingTokens" />

<!-- Emit token metrics with dimensions for cost attribution -->
<llm-emit-token-metric namespace="llm-metrics">
  <dimension name="API ID" value="@(context.Api.Id)" />
  <dimension name="Agent ID"
             value="@(context.Request.Headers.GetValueOrDefault("x-agent-id", "unknown"))" />
</llm-emit-token-metric>

The x-agent-id dimension is a small trick worth adopting: require each agent to identify itself in a header, and your token cost dashboard suddenly breaks down by agent instead of by subscription blob.

Verify the path end to end with a direct call:

curl -sS "https://apim-ai-gw.azure-api.net/llm/v1/chat/completions" \
  -H "Ocp-Apim-Subscription-Key: $APIM_KEY" \
  -H "Content-Type: application/json" \
  -H "x-agent-id: support-agent-01" \
  -d '{
    "model": "gpt",
    "messages": [{"role": "user", "content": "Summarize the SLA policy"}]
  }' | jq '.usage'

On the security side, the gateway authenticates to Azure-hosted model services with managed identity — no model API keys sitting in agent configuration — and can moderate prompts through Azure AI Content Safety policies before they reach the backend. OAuth flows for apps and agents accessing APIs or MCP servers run through APIM's credential manager.

The Reference Architecture

                    ┌──────────────────────────────┐
                    │      Azure API Management     │
                    │         (AI Gateway)          │
                    │                               │
   Agents ────────► │  Unified Model API (preview)  │
   MCP clients ───► │  MCP passthrough / exposure   │
   A2A peers ─────► │  A2A agent APIs               │
                    │                               │
                    │  llm-token-limit • semantic   │
                    │  cache • content safety •     │
                    │  token metrics • logging      │
                    └──────┬──────────┬─────────────┘
                           │          │
              ┌────────────┴──┐  ┌────┴───────────────┐
              │ Model backends │  │ Tool / agent layer │
              │ Foundry, AOAI, │  │ MCP servers, A2A   │
              │ Anthropic,     │  │ agents, internal   │
              │ Vertex, Bedrock│  │ REST APIs          │
              └───────────────┘  └────────────────────┘
                     ▲
                     │
              Azure API Center
              (inventory: APIs, MCP servers, agents)

Agents and MCP clients point at the gateway, not at backends directly. API Center holds the inventory so platform teams can answer what exists, who owns it, and whether it is approved.

Pitfalls I Keep Seeing

1. Anthropic Messages API requires the v2 tiers. If you run classic-tier APIM today and plan to front Claude through the gateway, budget for migration — this is documented as v2-only for now.

2. Prompt and completion logging is off by default. You enable it per API via diagnostic settings, and you should think hard before you do: prompts routinely contain personal data and secrets. Messages log in 32 KB chunks with a 2 MB cap per direction, so treat this as an auditing decision with a retention policy, not a default.

3. Semantic caching is extra infrastructure, not a checkbox. You need Azure Managed Redis or another RediSearch-compatible cache wired to APIM. Cache hits also mean stale answers — tune the similarity threshold and TTL deliberately for workloads where "close enough" responses are acceptable.

4. Token data is only as good as the backend response. Microsoft documents that when token usage is missing — broken streams, terminated connections — usage may be logged inaccurately or not at all. Do not build billing-grade chargeback on gateway logs alone; reconcile against provider meters.

5. A gateway only in front of the LLM is half a gateway. If agents can reach MCP tool servers and A2A peers directly, your governance has side doors. The point of this release is that MCP and A2A traffic can go through the same plane — use it.

6. Preview means preview. The unified model API and the Foundry integration are previews. Pilot them, design around them, but keep a fallback path until they GA.

When This Matters

This is worth your attention when you have multi-model strategies, agents connecting to several MCP servers, A2A chains, regulatory audit requirements, or token spend that needs attribution by team and workflow. For a single agent on a single Azure OpenAI deployment, the gateway fundamentals from my earlier article are sufficient — you can wait for this layer to mature.

Key Takeaways

  1. APIM now fronts the whole agent stack — LLM endpoints, MCP servers, and A2A agent APIs through one gateway, with inventory in Azure API Center.
  2. The unified model API (preview) is the headline bet: one OpenAI-compatible endpoint, automatic format translation, cross-provider routing and failover without code changes.
  3. Token governance is policy-level: llm-token-limit for quotas, semantic caching for cost and latency, llm-emit-token-metric with custom dimensions for per-agent cost attribution.
  4. Observability is real but bounded: prompts and completions log to Azure Monitor on opt-in, and token data can be incomplete on broken streams — reconcile before you charge back.
  5. Close the side doors: the value only lands if MCP and A2A traffic route through the gateway too, with managed identity instead of scattered API keys.

Continue the series: Azure API Management as AI Gateway — Managing Access, Throttling, and Observability for LLM Endpoints, Multi-Agent AI Systems in 2026 — Architecture Patterns (MCP, A2A, Swarm) for Production, and Copilot Studio vs Azure AI Foundry — The Enterprise Decision Framework.