What is an AI Gateway? Definition, Architecture, and How It Differs from an API Gateway

June 9, 2026 34 min read

ai-gateway architecture agent-governance proxy infrastructure

A team I talked to in early 2026 found out about their AI gateway problem the way most teams do: from a bill. Their product had quietly grown from one feature that called an LLM to nine of them. Each feature held its own provider key. Each one called the model directly. Then one feature shipped a retry loop with no ceiling, a model upgrade silently doubled the per-token price, and a prompt template change tripled the average input size. None of it was caught, because there was nothing in the middle watching. The monthly LLM bill went from a number nobody thought about to a number that needed a meeting. When the team tried to answer the obvious question, which feature spent what, the answer was not in any system they ran. It was in the provider’s billing console, aggregated, two days stale, and impossible to attribute.

That gap, the missing layer between your applications and the model providers, is what an AI gateway fills. This post defines the term precisely, walks the architecture, and draws the line between an AI gateway and the API gateway your platform team has run for years. They are not the same thing, they do not solve the same problems, and conflating them is how teams end up with the bill nobody could explain.

What is an AI gateway?

An AI gateway is a control layer that sits between your applications and one or more large language model providers, intercepting every model request and response to add routing, caching, cost control, observability, and security that the providers do not offer and that application code should not have to implement. It is the single point through which all LLM traffic flows, and the single place where the policy governing that traffic is defined and enforced.

The shape of the idea is familiar. A reverse proxy sits in front of web servers. A message broker sits between producers and consumers. An AI gateway sits between application code and model APIs. What makes it specific is not the proxy pattern, which is decades old. It is what the gateway understands about the traffic. A plain proxy forwards bytes. An AI gateway parses the LLM request: it knows the model name, it counts the tokens, it sees the prompt, it tracks the streamed response, it reads the cost. That semantic awareness is the whole point.

Three properties define an AI gateway and separate it from a generic proxy.

It is model-aware, not byte-aware. The gateway understands the protocol shape of an LLM call. It knows that model: claude-sonnet-4-6 is a routing decision, that messages carry a prompt that can be cached or inspected, that the response streams token by token, and that the request has a cost measured in input and output tokens rather than in bytes or requests. A normal API gateway sees an opaque POST body. An AI gateway sees an LLM request and can act on its contents.

It is the control point, not a helper library. An AI gateway is infrastructure that traffic passes through, not a convenience function the application calls. The application sends its request to the gateway endpoint. The gateway holds the real provider credentials, applies policy, and forwards the request to the actual model API. Because the gateway is in the network path rather than in the application process, its rules apply to every caller equally and cannot be skipped by a caller that forgets to import the right wrapper. This is the same distinction we drew in proxy vs SDK governance: a wall, not a door lock.

It is provider-agnostic by design. An AI gateway exposes one consistent interface and translates behind it. The application targets the gateway. The gateway routes to OpenAI, Anthropic, Google, a self-hosted model, or whatever the policy selects, and normalizes the differences. Adding a provider, switching a model, or splitting traffic across two becomes a gateway configuration change rather than an application code change. The application does not need to know which model answered.

In plain language: an AI gateway is the thing that turns “every feature calls the model directly and we hope for the best” into “all model traffic goes through one place we can see, price, route, and govern.”

In technical terms: an AI gateway is a protocol-aware reverse proxy for LLM and inference traffic that terminates provider-shaped requests (OpenAI Chat Completions, Anthropic Messages, and others), evaluates them against routing, caching, budget, and security policy, forwards allowed requests to a selected upstream model with the real credentials attached, and emits per-request observability keyed to token usage and cost.

How does an AI gateway differ from an API gateway?

An AI gateway differs from an API gateway in what it understands about the traffic. An API gateway governs HTTP-shaped concerns: routes, methods, headers, status codes, request rates. An AI gateway governs token-shaped, model-shaped, and prompt-shaped concerns: which model answers, how many tokens a call costs, what the prompt contains, whether a semantically similar answer is already cached. An API gateway is a necessary part of the stack. It is the wrong layer to govern LLM traffic, because the things that go wrong with LLM traffic are invisible at the layer it operates on.

This is the most common confusion in the category, so it is worth being precise. A platform team that has run Kong or AWS API Gateway for a decade will reasonably ask why the existing gateway cannot just handle the new AI endpoints. It can route them. It cannot govern them. The reason is structural.

An API gateway operates on the envelope. It reads the method, the path, the headers, the client identity, and the response status. It makes decisions in microseconds because it never has to look inside the body. A request is a request; the gateway counts it, rate-limits it, authenticates it, routes it, and forgets it. This model is correct for REST traffic, where every call is roughly equivalent and the cost of a call is dominated by the act of making it.

An LLM call breaks every assumption in that model. Two calls to the same endpoint can differ in cost by a factor of fifty, because one sent a 200-token prompt and the other sent a 12,000-token document. The cost lives in the body, not the envelope. The response is not a single status-coded payload; it streams token by token, sometimes for thirty seconds, and the API gateway’s request-counting and timeout logic was never designed for a response that arrives gradually. Caching is not a matter of matching a URL, because “summarize this contract” and “give me a summary of this contract” are different strings that should return the same answer. Rate limiting by request count is close to meaningless when one request can cost more than a thousand others combined. The unit that matters, the token, is something the API gateway cannot see and was never built to count.

The two gateways are complementary, not competing. The emerging consensus across vendors and practitioners is that an AI gateway does not replace an API gateway. A typical production stack runs both: the API gateway at the edge handling authentication, public REST routing, and coarse rate limiting, and the AI gateway behind it handling everything specific to model traffic. The AI gateway is the specialist. The API gateway is the generalist. Putting model traffic through only the generalist leaves the expensive, model-specific failure modes ungoverned.

The table below makes the split concrete. The left column is the world an API gateway was designed for. The right column is the world an AI gateway has to handle.

Concern	API gateway	AI gateway
Primary unit	HTTP request	LLM call measured in tokens
What it inspects	Method, path, headers, status	Model name, prompt, token counts, streamed response
Cost model	Roughly uniform per request	Varies 50x or more per request by prompt size and model
Rate limiting	Requests per second	Tokens per minute, spend per day, per agent or tenant
Caching	Exact URL and query match	Semantic match on prompt meaning
Routing decision	Path to backend service	Model selection by cost, latency, capability, availability
Response shape	Single status-coded payload	Token-by-token stream, often multi-second
Failover trigger	HTTP 5xx from backend	Provider error, rate limit, latency, model unavailable
Security focus	Endpoint access, request validation	Prompt content, PII egress, injection patterns, output rules
Observability key	Endpoint, status, latency	Model, tokens, cost, cache hit, per-feature attribution
Latency budget	Microseconds	Milliseconds, negligible next to model inference

The pattern across the table is consistent. Everything the API gateway governs lives in the envelope and is cheap to inspect. Everything the AI gateway governs lives in the body and requires parsing the LLM request to act on. That is the entire difference, and it is why one gateway cannot quietly absorb the other’s job.

What functions does an AI gateway provide?

An AI gateway provides seven functions that recur across every serious product in the category: model routing, semantic caching, token metering, cost controls, a prompt firewall, observability, and failover. Not every gateway implements all seven, and the depth varies, but this is the function catalog the term implies. Each one exists because a specific class of LLM-traffic problem has no good home in application code or in an API gateway.

Model routing

Model routing is the gateway selecting which model answers a given request, based on policy rather than a hardcoded choice in the application. The application asks for “a model.” The gateway decides which one, and can decide differently per request, per tenant, or per traffic split.

Routing solves several problems at once. It lets you send cheap, high-volume requests to a small model and reserve a frontier model for the requests that need it, without the application tracking which is which. It lets you run an A/B test between two models by splitting traffic at the gateway. It lets you migrate off a deprecated model by changing one config line instead of redeploying every service. Portkey and LiteLLM both expose conditional routing, load balancing across providers, and weighted splits as core configuration.

routing:
  default: claude-haiku-4
  rules:
    - when: { feature: "contract-analysis" }
      route_to: claude-sonnet-4-6
    - when: { tenant_tier: "free" }
      route_to: gpt-4o-mini
    - when: { request_tokens: ">8000" }
      route_to: claude-sonnet-4-6
  split:
    - model: claude-sonnet-4-6
      weight: 90
    - model: gpt-5
      weight: 10

Semantic caching

Semantic caching is the gateway returning a stored response when a new prompt is close enough in meaning to a previous one, rather than calling the model again. The match is on meaning, not on exact text. “Summarize this document” and “give me a summary of this document” are different strings and would never hit a URL-match cache, but they are the same request, and a semantic cache returns the same answer for both.

The mechanism is an embedding plus a vector similarity search. The gateway embeds the incoming prompt, searches a vector store for prior prompts above a similarity threshold, and on a hit returns the cached completion without touching the model. Kong’s AI Semantic Cache plugin reports responses up to 20 times faster on a cache hit, because a vector lookup is far cheaper than model inference. Semantic caching is distinct from prompt caching, which the providers offer to discount repeated prefix tokens; we cover the difference in prompt caching vs semantic caching. The honest caveat: semantic caching trades exactness for speed and cost. A similarity threshold set too loose will serve a near-miss answer. This is a tuning decision, and it is covered in the limits section below and in semantic caching for AI agents.

Token metering

Token metering is the gateway counting the input and output tokens of every call and attributing them to a caller. This is the function that the team in the opening incident did not have. Without it, spend is a single provider-level number. With it, spend is a per-feature, per-tenant, per-agent, per-model breakdown that updates in real time rather than two days late.

Metering is the foundation for almost everything else. Cost controls need a token count to enforce against. Observability needs it to attribute spend. Routing decisions that optimize for cost need it to know what each route costs. A gateway that meters tokens turns “the AI bill” into a set of numbers a finance team can actually allocate, which is the subject of AI agent cost attribution.

Cost controls

Cost controls are budget ceilings the gateway enforces at request time, before the call reaches the provider, so a runaway loop or a misconfigured job stops at a known limit instead of at the next billing cycle. This is the difference between finding out about a cost problem from a dashboard alert and finding out from an invoice.

budgets:
  - scope: { feature: "auto-summarize" }
    daily_usd: 50.00
    on_exceed: block
  - scope: { tenant_tier: "free" }
    monthly_usd: 5.00
    on_exceed: downgrade_model
  - scope: { agent: "research-agent" }
    per_request_max_tokens: 20000
    loop_detection:
      window: 60s
      max_identical_requests: 5
      on_trip: block_and_alert

The reason this belongs in the gateway and not in the application is the same reason metering does. An application enforcing its own budget is checking a limit inside the process that holds the key and runs the loop. The gateway enforces from outside that process. A surprise four-figure bill from a retry loop is precisely the case where the thing being limited cannot be trusted to limit itself. We made this argument in full in cut AI API costs without code changes.

Prompt firewall

A prompt firewall is the gateway inspecting prompt content and model output for things that should not pass: personally identifiable information, credentials, known prompt-injection patterns, and policy-restricted content. Because the gateway parses the LLM request body, it can see the prompt text, which an API gateway cannot.

Cloudflare AI Gateway’s Data Loss Prevention scans both the prompt going to the model and the response coming back, flagging or blocking sensitive data patterns. Kong’s Semantic Prompt Guard blocks categories of prompt by meaning rather than by keyword, and Kong reports built-in PII detection across 20 categories. The firewall is a content filter at the chokepoint, and it is reliable there for the same structural reason every gateway function is reliable there: every prompt goes through one place.

Observability

Observability is the gateway emitting a per-request record keyed to the things that matter for LLM traffic: model, token counts, cost, latency, cache hit or miss, and the caller. A normal request log tells you an endpoint returned 200 in 800 milliseconds. An AI gateway log tells you that feature X called claude-sonnet-4-6 with 4,200 input tokens, streamed 900 output tokens, cost 7.4 cents, missed the cache, and took 3.1 seconds. That is the record you need to debug a latency regression, attribute a cost spike, or measure a cache hit rate.

Failover

Failover is the gateway retrying a failed call against a different model or provider, automatically, so a provider outage or rate limit degrades quality instead of breaking the feature. Portkey’s fallback chains trigger on any non-2xx response by default and can be tuned to specific codes such as 429 and 503, walking a prioritized list of models until one answers. The application sees a slightly slower or slightly different response. It does not see an error, and it does not need failover logic of its own.

Where does an AI gateway sit in the architecture?

An AI gateway sits between application code and model providers, in the network path, so that every model request passes through it before reaching a provider and every response passes back through it before reaching the application. It holds the real provider credentials. The application holds only a credential that authenticates against the gateway. There is no path from the application to a model that does not go through the gateway, because the application has nothing that would authenticate anywhere else.

The placement is the design. If the gateway were a library the application imported, a caller that skipped the import would skip the governance. If the gateway were optional, it would be governance by convention, and convention is not enforcement. Putting the gateway in the network path and giving it sole custody of the provider keys makes it the only door. The reference architecture below shows the position.

Two deployment shapes are common. The gateway can run as a standalone service that all applications point at, which is the simplest model and the one most teams start with. Or it can run as a sidecar on the same host or pod as the application, which removes the network round trip and is used on latency-sensitive paths. Either way the principle holds: the application’s configuration points at the gateway, the gateway’s configuration holds the provider keys, and the two are deliberately separate so that the governance cannot be skipped by a caller and the keys cannot be extracted by one.

There is a relationship worth naming here. An AI gateway and an agent policy engine overlap. Both sit in the path, both hold credentials, both enforce policy. The distinction is one of emphasis. An AI gateway is organized around model traffic: routing, caching, token economics, provider abstraction. A policy engine is organized around authorization: deny verdicts on tool calls, approval workflows, tenant isolation, audit-grade decision logs. In practice a mature product does both, and a gateway that adds tool-call governance and a policy engine that adds model routing converge on the same component. The two terms describe the same chokepoint from two angles.

Do you need an AI gateway if you use one provider?

If you use exactly one provider and one model, you do not need an AI gateway for routing or provider abstraction, because there is nothing to route between and nothing to abstract. You may still need it for everything else: cost control, metering, semantic caching, prompt firewall, observability, and failover. Most of the value of an AI gateway is not multi-provider. The multi-provider story is the most visible feature, but it is not the load-bearing one.

This is worth being honest about, because the single-provider case is where the “do I need this” question is sharpest. Here is the straight answer. Provider abstraction and cross-provider routing genuinely do not apply if you have one provider. Strike those off the list. But the team in the opening incident used one provider, and their problem was not routing. Their problem was that nine features shared keys, nobody could attribute spend, and a retry loop ran unchecked. Every one of those problems exists with one provider. Token metering, per-feature cost attribution, budget ceilings, loop detection, semantic caching, PII inspection, and unified observability are all single-provider problems that an AI gateway solves and that application code solves badly.

There is also a failover argument that survives the single-provider case. One provider still has outages, still rate-limits, and still deprecates models. A gateway with one provider configured can still fail over between models within that provider, can still queue and retry on a 429, and can still let you migrate off a deprecated model with a config change. The single-provider failover surface is smaller than the multi-provider one. It is not zero.

The case where you genuinely do not need an AI gateway: a single developer, a single feature, a single model, low and predictable spend, no compliance requirement, no untrusted input. At that scale the gateway is overhead. The honest framing is the one Mark uses for every infrastructure decision. We think the gateway earns its place the moment you have more than one feature calling a model or more than one person who would have to answer for the bill, and not before. Below that line, direct calls are fine. Above it, the question is when, not whether.

AI gateway vs SDK wrapper vs direct calls

There are three ways to connect application code to a model: call the provider directly, wrap the provider client in an SDK governance library, or route through an AI gateway. They are not equivalent, and the right choice depends on stakes, scale, and how much you can trust the calling code.

Direct calls are the default and the simplest. The application imports the provider SDK, holds the API key, and calls the model. There is nothing in the path. This is correct for a prototype, a single feature, or a script. It has no governance, no metering, no caching, and no failover, and every one of those becomes your problem to hand-build the moment you have more than one caller.

An SDK wrapper is a governance library the application imports around the provider client. It can add budget checks, logging, and retries inside the process. It is faster to adopt than a gateway because it is a dependency change rather than an infrastructure change. Its limit is structural and we covered it in depth in proxy vs SDK governance: the wrapper runs inside the process that holds the key, so any caller that imports the raw client, spawns a subprocess, or makes a direct HTTP call goes around it. An SDK wrapper is a door lock. It governs the callers that agree to use the door.

An AI gateway is infrastructure in the network path. It governs every caller because it holds the only credentials that work and there is no other door. It is a wall. The cost is that it is an infrastructure component to deploy and operate, and it adds a network hop. For production traffic, multiple callers, autonomous agents, or anything with a compliance or cost requirement, that cost is worth paying. For a prototype it is not.

Factor	Direct calls	SDK wrapper	AI gateway
Setup cost	None	Low, a dependency	Moderate, deploy a service
Enforcement boundary	None	Inside the process	Outside the process
Bypassable by the caller	N/A	Yes, several ways	No
Multi-language support	Per SDK	Per language	Yes, HTTP level
Covers autonomous agents	No	Weakly	Yes
Provider abstraction	No	Sometimes	Yes
Right for	Prototype, one feature	Solo dev, trusted single-process app	Production, multiple callers, agents, compliance

The three are not strictly a ladder. A reasonable production setup runs an SDK wrapper for rich in-process observability and an AI gateway for hard enforcement, the same hybrid we recommended for proxy and SDK governance. The wrapper sees the most detail. The gateway makes the rules stick.

What are the limits of an AI gateway?

An AI gateway is a control layer, not a fix for everything that goes wrong with LLM traffic. It adds a network hop, it concentrates risk in one component, and it does nothing about the quality of what the model is asked or what it answers. Honest engineering means naming these limits before a reader hits them in production.

An AI gateway adds latency. Every request now crosses an extra component. A well-built gateway adds single-digit to low-double-digit milliseconds for policy evaluation, routing, and a cache lookup. Against model inference, which runs from a few hundred milliseconds to several seconds, that overhead is usually unmeasurable. On a latency-critical path it still has to be designed for: run the gateway as a sidecar to remove the network round trip, keep the policy set tight, and cache routing decisions. The overhead is small. It is not zero, and pretending otherwise is dishonest.

An AI gateway is a single point of failure. Everything routes through it, so if it is down, all model traffic is down. This is the unavoidable trade of any chokepoint. It is managed the way every critical proxy is managed: run the gateway as a clustered, health-checked service with failover, not a single instance. Concentrating traffic into one component is acceptable only when that component is built to be highly available, and the gateway must fail in a defined way, not an arbitrary one.

An AI gateway does not fix bad prompts. It routes, caches, meters, and inspects. It does not improve a prompt that is poorly written, and it cannot make a model reason better. If a feature gives the model a confusing instruction, the gateway faithfully forwards the confusing instruction. Prompt quality, evaluation, and output validation are real disciplines, and they live above the gateway, not inside it.

An AI gateway does not catch every bad output. The prompt firewall is a pattern-and-classifier filter. It is good at structured leaks: PII patterns, credential formats, known injection strings. It is not a guarantee that nothing harmful or wrong passes. A model can produce a confidently incorrect answer that violates no pattern, and the firewall has no basis to block it. Output correctness needs evaluation and, for high-stakes paths, human review. The gateway reduces a class of risk. It does not eliminate the category.

Semantic caching trades exactness for cost. A semantic cache returns a stored answer when a new prompt is close enough in meaning. Set the similarity threshold too loose and it returns a near-miss answer to a question that deserved a fresh one. This is a tuning decision with a real failure mode, and for prompts where the answer must be exact, semantic caching should be scoped narrowly or turned off. Speed and cost on one side, exactness on the other; the gateway gives you the dial, not the right setting.

The honest summary: an AI gateway is the right home for routing, token economics, caching, and traffic-level security, and it is the only place those can be enforced for every caller at once. It is not a substitute for prompt engineering, for model evaluation, for output review, or for high availability engineering. It is one layer. It composes with the others.

How do you adopt an AI gateway?

You adopt an AI gateway by inserting it into the path in observe-only mode first, turning on the cheap high-value functions before the strict ones, and moving to enforcement only after the observability has told you what your traffic actually does. Adoption is a migration, not a switch, and rushing to default-deny enforcement on day one breaks features that were quietly depending on no governance at all.

The migration path has four stages.

Stage one: route traffic through the gateway in observe-only mode. Point your applications at the gateway. Move the provider keys into the gateway and out of the applications. Configure the gateway to forward every request and block nothing. The only thing turned on is observability. Run this for one to two weeks. The decision log will show you what every feature actually calls, with what model, at what token cost, which is almost always different from what the team believes.

Stage two: turn on metering and observability. With traffic flowing through the gateway, per-feature and per-tenant token attribution is now available. This is the function that pays for the adoption on its own, because it converts the AI bill from one number into an allocable set of numbers. Nothing here blocks a request, so there is no risk to features. This is the safe, immediate win.

Stage three: turn on the low-risk functions. Enable semantic caching with a conservative similarity threshold, and enable failover chains. Caching reduces cost and latency on a cache hit. Failover increases resilience. Neither denies a request, so the blast radius of a misconfiguration is a cache miss or an extra retry, not a broken feature.

Stage four: turn on enforcement. Now enable budget ceilings, loop detection, and the prompt firewall. Start with generous limits and warn-only modes, watch the decision log, then tighten. This is the stage where a request can be blocked, so it goes last, after the observability from the earlier stages has told you where the real limits should sit.

The first capabilities to turn on, in order, are metering, then caching and failover, then budgets and the firewall. The principle is to lead with the functions that cannot break a feature and finish with the functions that can. The team in the opening incident needed exactly stage two: they needed to see the spend before they could control it. Most teams find that the observability alone justifies the project, and the enforcement is what they were always going to want once they could see clearly enough to set the limits.

FAQ

What is an AI gateway in simple terms?

An AI gateway is a single piece of infrastructure that all of your application’s calls to AI models pass through. It sits between your code and the model providers. Because every model request crosses it, it is the one place you can see what is being spent, route requests to the right model, cache repeated answers, enforce budgets, and inspect prompts for sensitive data. Without it, every feature calls the model directly and there is nothing in the middle watching. With it, model traffic becomes something you can see and govern in one place.

Is an AI gateway the same as an API gateway?

No. An API gateway governs HTTP-shaped concerns: routes, methods, headers, authentication, and request rates. It treats the request body as opaque. An AI gateway governs the contents of an LLM request: the model chosen, the tokens consumed, the prompt text, and the streamed response. An API gateway counts requests; an AI gateway counts tokens, and tokens are the unit that determines cost. The two are complementary. A production stack typically runs an API gateway at the edge and an AI gateway behind it for model traffic. The API gateway cannot do the AI gateway’s job because the things that go wrong with LLM traffic, runaway token cost, the wrong model answering, prompt leakage, are invisible at the layer an API gateway operates on.

Do I need an AI gateway if I only use one LLM provider?

Often yes, but not for routing. Cross-provider routing and provider abstraction genuinely do not apply with one provider. Everything else still does. Token metering, per-feature cost attribution, budget enforcement, loop detection, semantic caching, prompt inspection, and unified observability are all single-provider problems, and an AI gateway solves them in one place where application code solves them badly and inconsistently. A single provider still has outages, rate limits, and model deprecations, so even failover has a smaller but real role. The case where you do not need one is a single developer with a single feature and low predictable spend. Above that, the gateway earns its place.

What is the difference between an AI gateway and an LLM gateway?

In current usage the two terms are interchangeable. Both describe a control layer between applications and large language model providers that adds routing, caching, metering, cost control, and security. “LLM gateway” is the more literal name, since the traffic is LLM calls. “AI gateway” is the more common name and the one most vendors, including Cloudflare and Kong, have settled on. Some vendors use “AI gateway” to suggest a broader scope that includes non-LLM inference such as embeddings, image models, or speech, but there is no firm industry line. If a product is described as either, expect the same core function catalog.

How much latency does an AI gateway add?

A well-built AI gateway adds single-digit to low-double-digit milliseconds per request for policy evaluation, routing, and a cache lookup. Model inference takes from a few hundred milliseconds to several seconds, so the gateway overhead is usually a fraction of a percent of the total response time and not measurable in practice. On a semantic cache hit the gateway makes the request dramatically faster, since a vector lookup is far cheaper than calling the model. On latency-critical paths the gateway can be deployed as a sidecar on the same host as the application, which removes the network round trip entirely. The overhead is real but small, and on cache hits it is strongly negative.