Picture a familiar rollout: a team wires their agent framework to a single AI gateway, expecting one deployment to solve model routing, cost tracking, and tool governance at once. Six weeks later, the model traffic is centralized and clean. The MCP servers the agents call are still wired in ad hoc, one Claude MCP add at a time, because the gateway that promised to cover “LLMs and agents” turned out to cover LLMs well and agents in name only.
That gap isn’t a product failure. It’s a category that got named broadly before the two things inside it were built with equal depth. An AI gateway in most of today’s market is fundamentally a model-routing layer that added MCP support as a feature line, not a second discipline. Knowing which half of that promise a given product actually delivers is the decision that matters before anything else.
AI Gateway vs API Gateway: Two Different Traffic Problems
Strip the marketing language, and an AI gateway does one of two jobs, sometimes both, rarely both equally well. The first job is LLM gateway work: a single endpoint that routes requests across model providers, OpenAI, Anthropic, and Bedrock, and returns a consistent response format regardless of which large language model provider actually served it. The second job is governing what an AI agent does once a model decides to call a tool, which is a model context protocol problem, not a model-routing one.
Those two jobs get bundled under one label because they share infrastructure patterns, a proxy in front of outbound AI traffic, centralized credentials and request logging, but the actual work each one does is structurally different. As AI systems move from single-model pilots to production AI infrastructure spanning multiple AI models and diverse AI services, that bundling gets more visible, not less, because the volume of AI consumption an organization has to govern grows on both sides of the split at once. A traditional API gateway manages request volume and authentication for conventional web services. An AI gateway extends that pattern for AI workloads specifically: token-denominated billing instead of per-request billing, streaming responses instead of single round trips, and a threat surface that includes prompt injection and data leakage, not just malformed requests. Routing a chat completion request to whichever of several AI providers is cheapest or fastest right now has nothing in common with deciding whether an AI agent’s identity is scoped to write to a production database. Products that lead with the first job and mention the second as a feature line usually have that difference baked into how thin their MCP coverage actually is.
Credential Management and Cost Control: What the LLM Routing Layer Already Solves
The core capabilities here are well established across the market and worth naming precisely, because they’re genuinely useful and genuinely not the same thing as agent governance.
Unified API normalizes provider-specific SDKs into one standardized interface, so switching from one model to another is a configuration change, not a code change. Credential management keeps provider API key material out of application code entirely, often issued as virtual keys scoped to a specific team or budget rather than the raw provider credential, with every request authenticating to the gateway instead of the underlying provider. Traffic splitting and fallback chains route a percentage of requests to a new model for comparison or trigger model failover to a backup provider automatically when the primary one is down, with load balancing distributing requests across multiple deployments of the same model. Token-based rate limiting enforces spend ceilings against the actual billing unit for LLMs, tokens, rather than the raw request count a traditional API gateway would throttle. Usage and cost tracking attributes token usage per team, per endpoint, or per project, which is the difference between discovering a spend problem in a monthly bill and catching it the week it started. Semantic caching goes further, matching a new prompt against previously cached responses by vector similarity rather than exact text, so semantically identical questions worded differently still hit the cache instead of triggering a new model call.
Enhanced Security and Compliance Are Already Table Stakes Here
The enhanced security layer here is mature too, and worth naming precisely rather than waving at generically. A serious gateway enforces encryption for data in transit and at rest by default, not as an add-on, and pairs that with real-time network monitoring for anomalous request patterns rather than after-the-fact log review. Prompt management, storing and versioning the actual prompt templates an application sends, keeps that logic centralized instead of scattered across application code, where a change means a redeploy. Most of this same layer also maps directly to compliance frameworks like GDPR, since centralized logging and access control are most of what a regulatory audit actually asks for. None of it is unique to any one vendor. It’s the baseline the routing half of the category has largely converged on.
Cost control rounds out the routing layer’s maturity. Real-time dashboards built on usage tracking and usage metrics turn spend from a monthly surprise into something a team can watch as it happens, and cost management at that granularity is what lets an organization allocate model spend across teams the same way it already allocates cloud spend, rather than treating AI consumption as one undifferentiated line item.
None of this touches what happens once a model decides to call a tool. A gateway that routes a completion request perfectly can still hand that completion to an agent with no visibility into which MCP server it then reaches, what credential it uses to reach it, or what it does once it’s there. Products built primarily for this layer take different bets on what to optimize: LiteLLM leads on open-source provider breadth and self-hosted flexibility, Portkey on per-request observability and tracing for application teams, Bifrost on raw routing throughput and low gateway overhead, and Envoy AI Gateway on extending an already-established proxy, built on the same Envoy core that underlies much of existing service-mesh infrastructure, into LLM-specific routing. Reading their own documentation on MCP coverage against their documentation on the thing they actually optimized for makes the depth gap visible quickly.
Where MCP Governance Actually Lives
This is the layer most “AI Gateway” products treat as a checkbox rather than a discipline, and it’s worth being specific about what real coverage here requires, because the gap is where production incidents actually happen.
Server discovery and cataloging answers which MCP servers exist, who approved them, and what tools they expose, before an agent ever connects to one. The open-source MCP Gateway and Registry project treats this as a distinct capability from routing, enforcing fine-grained access control at discovery time so a restricted server never shows up in a search result for an identity that shouldn’t see it. That enforcement is a form of role-based access control, but scoped to individual tools and servers rather than the coarser application-level roles most identity systems already manage.
Tool-call level policy enforcement decides whether a specific action, not just a connection, is authorized, and it’s a different check than whether a model completion request is authenticated. A gateway that authorizes a connection but doesn’t scope individual tool calls is making a connection-level promise about a call-level risk, and that gap matters most when the tool in question can touch sensitive data. Since the 2026-07-28 MCP specification, method and tool names travel in HTTP headers (Mcp-Method and Mcp-Name), so a gateway can check each call against policy without parsing the request body, which leaves even less reason to stop at the connection. Data masking at the tool-response level, redacting fields before they ever reach the model’s context window, is a related but separate control most AI Gateway products document for LLM prompts and skip entirely for MCP tool outputs.
Registration-time scanning vets a server’s tool definitions before an agent can reach them at all. The same AWS-backed registry runs three open-source Cisco AI Defense scanners against every asset at registration, disabling anything that fails before it’s ever discoverable, which is the clearest publicly documented example of this checkpoint done openly rather than behind a closed detection model.
Runtime threat detection watches for the failure modes the OWASP Top 10 for Agentic Applications formalized in December 2025: tool descriptions rewritten after approval, responses carrying injected instructions, an agent’s call pattern drifting outside its approved scope. This is continuous, not a one-time check at connection.
CVE-2026-32211, a missing-authentication flaw Microsoft rated CVSS 9.1 in the Azure MCP Server (NVD scored it 7.5), is a concrete example of why all four of these matter as separate checks. A gateway routing model traffic correctly, authenticating every completion request cleanly would still route agent traffic straight through that vulnerable server, because none of that routing logic was ever built to independently vet a server’s own security posture.
The honest reason most AI Gateway products cover one layer deeply and the other thinly is architectural, not a resourcing failure. A gateway built for high-throughput model routing optimizes for low added latency on a simple request-response pattern: authenticate, route, return. MCP’s protocol layer became stateless in the 2026-07-28 specification, but agent traffic still doesn’t follow that pattern. Agents discover tools dynamically, chain calls across multiple servers based on model reasoning rather than a fixed routing table, and take actions with downstream consequences a completion request never has. Building deep governance for that pattern is a different engineering problem than shaving milliseconds off provider routing, and a team that optimized hard for the first rarely has the same depth left for the second.
That’s not an argument against unified platforms. It’s an argument for checking which half of the platform your actual risk sits in before assuming both are covered equally, and for reading a vendor’s own documented feature depth rather than the category label on the page. A product page that spends several paragraphs on traffic splitting and fallback chains, then covers MCP governance in a single paragraph without naming a registry, a per-tool policy model, or a registration-time vetting step, is telling you exactly where its engineering investment actually went. That’s not a criticism specific to any one vendor. It’s the pattern across most of the category right now, because the routing problem got solved first and the governance problem is still catching up.
What to Evaluate for Each Layer
For the LLM routing half: provider coverage and failover behavior under an actual outage, not just a documented feature list; whether cost attribution is granular enough to catch a spend problem before the invoice does; whether switching providers is genuinely a config change or requires touching application code.
For the MCP governance half: whether server discovery and cataloging exist as a distinct capability from routing, not folded into a single “MCP support” checkbox; whether policy is enforced per tool call or only at the connection level; whether registration-time vetting happens before a server is reachable, or only after something has already gone wrong with it; whether the audit trail records what a tool call actually did downstream, not just that a call was made; and whether the product can see MCP servers configured on developer machines that never route through it at all.
A product that’s honest about which half it covers deeply, and clear about where the other half needs a second layer, is more useful than one that claims both without documenting either in depth.
Where Obot Fits Once Your AI Agents Need More Than Routing
Obot treats the second layer as an Enterprise AI Control Plane built around three jobs: define what agents can reach, choose where they run, and prove what they did.
Gateway: Define What Agents Can Reach
Obot MCP Gateway is not a model router. It doesn’t compete on provider failover, traffic splitting, or cost dashboards across LLM vendors, and it won’t pretend otherwise. What it does is the second layer this article has spent most of its length on: a centralized control plane for MCP specifically, with a built-in catalog for server discovery, granular access control enforced at the individual tool level rather than the server level, and an audit trail that operates at the tool-call level regardless of what routing layer sits in front of it.
Obot is open-source under the MIT license, self-hostable on Kubernetes or Docker, or available as a managed service, same product either way, with enterprise identity federation through Okta and Microsoft Entra in the Enterprise edition. Self-hosting matters specifically for organizations where data sovereignty rules out routing agent traffic through third-party infrastructure, a constraint that applies as much to artificial intelligence tool governance as it does to any other regulated system. For organizations already running an LLM gateway for model traffic, Obot sits beside it rather than replacing it, governing the MCP layer that most AI Gateway products document as a feature line and treat as an afterthought.
Device: Watch and Enforce With Obot Sentry
The opening example ended with MCP servers wired in one Claude MCP add at a time. Those servers live in local client configuration files, and a local server that talks straight to an internal system never passes through a gateway of either kind. Obot Sentry reads those configurations across Claude Code, Claude Desktop, Codex, Cursor, and VS Code, reports which MCP servers, skills, and plugins each machine loads, and where enforcement is on, and fails closed on servers that aren’t approved. Most products in this category, routing-first or MCP-first, govern only the traffic that reaches them.
Hosted: Contain Agents in Governed Environments
For agents that shouldn’t run on a laptop at all, Obot hosted environments run coding agents such as Claude Code under the same control plane, so their connections come from policy rather than a local config file.
Where This Leaves You
“AI Gateway” is a real category with real infrastructure value on both halves of what it promises. It is not, in most of today’s market, one product doing both jobs to the same depth. The routing half is mature and well understood. The MCP governance half is where the actual incidents happen, and it’s the half worth checking for real depth before assuming a single deployment has closed the gap
