Multi-Provider AI: Failover, Routing, and Cost Control
A single LLM provider is a single point of failure and a single price you cannot negotiate. Running multiple providers buys resilience and cost control, if you handle the routing right.
The morning every AI feature broke at once
On November 18, 2025, a configuration file inside Cloudflare’s network grew past an expected size and crashed the software that handles traffic for a large slice of the internet. The outage ran about five and a half hours. ChatGPT and Sora went down. Claude reported issues. X, Discord, Spotify, and Shopify all took the hit. Cloudflare routes traffic for roughly 20% of the world’s websites, so when its Workers KV storage platform failed, the failure did not stay contained (SiliconANGLE coverage, Cloudflare’s own incident analysis).
If your product called one model provider that morning, your AI feature was down too. Not because of a bug in your code. Because of a database bug two infrastructure layers below you, in a company you may not have known you depended on.
That morning was not an outlier. It was the loudest entry in a long list. OpenAI’s status page logs a 4.5-hour outage on December 11, 2024 traced to a server configuration mistake, and a roughly 9-hour outage on December 26, 2024 caused by a power failure at a Microsoft Azure data center (OpenAI status, Storyboard18 outage timeline). Anthropic’s status history shows a widespread incident on December 19, 2025 with elevated error rates on Claude Sonnet 4.5, where Downdetector complaints peaked above 6,800 per hour (Anthropic status history). Providers publish uptime numbers around 99%. That sounds reassuring until you do the arithmetic: 99% uptime is more than three and a half days of downtime a year. If all of that downtime lands during your business hours, on one provider, your users feel every minute of it.
The fix is not a better provider. There is no provider that does not have bad mornings. The fix is to stop betting your entire AI feature on one of them. This post is about how to run AI across multiple providers: Anthropic, OpenAI, Google, and open models. Failover so an outage degrades your service instead of stopping it. Routing so you pay the right price for each request. And an honest accounting of what multi-provider costs you in return, because it is not free.
Why depend on more than one AI provider?
Single-provider is the default because it is simple. One SDK, one API key, one bill, one set of docs. Simple is a real virtue and you should not give it up without reasons. There are four.
Resilience. This is the headline reason and the Cloudflare morning made it concrete. When your only provider is unreachable, your only AI feature is unreachable. With a second provider wired in, an outage becomes a routing event instead of an incident. Traffic shifts, latency rises slightly, and your users keep working. The outage still happened. Your product just did not have to care.
Cost. Provider prices for comparable capability are not the same, and they move. As of mid-2026, Claude Sonnet 4.6 lists at $3.00 per million input tokens and $15.00 per million output. GPT-5.4 lists at $2.50 input and $15.00 output. Gemini 2.5 Pro lists at $1.25 input. Gemini 2.5 Flash sits at $0.30 input, and Gemini 3.1 Flash-Lite is among the cheapest proprietary options at roughly $0.10 input and $0.40 output. DeepSeek V3.2 lands near $0.14 input (pricepertoken.com, CloudZero pricing comparison). A request that genuinely needs a frontier model should get one. A request that needs a sentence classified should not pay frontier rates. If you are locked to one provider, every request pays that provider’s price for that capability tier whether the task warrants it or not. We covered the single-provider version of this in cutting AI API costs without code changes; multi-provider extends the same logic across vendors.
Capability coverage. No single provider is best at everything. One model handles long-context document analysis well. Another is stronger at code generation. A third is cheap and fast enough for high-volume classification. A fourth runs open-weight and can be self-hosted for data that cannot leave your network. Locking to one provider means locking to its weakest capability as much as its strongest.
Negotiating leverage. A provider that knows you cannot leave has no reason to give you a better rate, a higher rate limit, or priority support. A provider that knows 40% of your traffic could move to a competitor next quarter has every reason. Multi-provider capability is leverage even in the months you do not exercise it.
Here is the mid-2026 picture across the four provider categories, with list pricing per million tokens. Prices move, and they move fast: industry list prices fell roughly 80% from 2025 to 2026 (pricepertoken.com, TLDL pricing roundup). The point of the table is not the exact numbers, which will be stale within a quarter. It is the spread. The cheapest input tier is more than an order of magnitude below the most expensive, and no single provider owns every column.
| Provider | Representative model | Input / 1M | Output / 1M | Strength | Where it fits |
|---|---|---|---|---|---|
| Anthropic | Claude Sonnet 4.6 | $3.00 | $15.00 | Reasoning, code, long context | Tier-three primary |
| OpenAI | GPT-5.4 | $2.50 | $15.00 | Broad capability, tooling | Tier-two and tier-three |
| Gemini 2.5 Pro | $1.25 | $10.00 | Long context, price-to-capability | Tier-two primary | |
| Gemini 3.1 Flash-Lite | $0.10 | $0.40 | Cheap, fast, high volume | Tier-one primary | |
| Open model | DeepSeek V3.2 | $0.14 | $0.28 | Cheapest, self-hostable | Tier-one fallback, private data |
Resilience, cost, coverage, leverage. Notice that only the first one is about outages. The other three apply on a perfectly calm day when every provider is up. That matters, because the work of going multi-provider is the same whether or not you ever see another outage, and three of the four payoffs are collecting the whole time. A team that wires in a second provider purely for the outage insurance, then never builds routing, has still left the cost, coverage, and leverage payoffs on the table. They are real money, and they do not require a single provider to ever have a bad day.
What breaks when you go multi-provider?
Here is the founder-honest part, up front, before the how-to. Multi-provider is not a config flag. The reason most teams stay single-provider is not ignorance. It is that providers are not interchangeable, and the differences are exactly the kind that do not show up in a quick demo and do show up in production at 2am.
API shape differences. Anthropic’s Messages API, OpenAI’s Chat Completions API, and Google’s Gemini API are three different request and response formats. They disagree on how system prompts are passed, how multi-turn history is structured, how tool definitions are declared, how streaming events are framed, and what an error response looks like. Code written against one does not run against another. Something has to translate.
Prompt portability. A prompt tuned for one model is not a prompt tuned for another. Few-shot examples that steer GPT cleanly can confuse Claude. A system prompt that Gemini follows precisely might get loosely interpreted by a smaller open model. Output formatting instructions land differently. The prompt is not portable data; it is calibrated to a specific model’s behavior, and moving it costs re-testing.
Token counting differences. Each provider uses its own tokenizer. The same paragraph of English is a different token count to Anthropic, OpenAI, and Google. This breaks naive cost estimates, breaks context-window math if you assume one number across providers, and breaks any routing rule that triggers on a token threshold unless you count per-provider.
Feature parity gaps. Prompt caching works differently on every provider, and as we detailed in prompt caching vs semantic caching, the activation method, minimum token counts, cache duration, and discount all vary by vendor. Structured output modes, tool-use formats, vision support, context window sizes, and reasoning controls are all uneven. If you build a feature on one provider’s capability, that feature does not automatically exist on your fallback provider.
The table below makes the divergence concrete. Each row is a place where a request that works on one provider needs translation, retuning, or a fallback plan to work on another.
| What differs | Anthropic | OpenAI | |
|---|---|---|---|
| API format | Messages API | Chat Completions API | Gemini API |
| System prompt | Top-level system field | A message with role: system | systemInstruction field |
| Tool definitions | tools with input schema | tools with function schema | functionDeclarations |
| Tokenizer | Anthropic tokenizer | OpenAI tokenizer | Gemini tokenizer |
| Prompt caching | Explicit breakpoints or auto flag | Fully automatic | Implicit or explicit API |
| Error shape | error.type plus message | error.code plus type | error.status plus details |
A request is not a portable object you move between these columns. It is a thing built for one column. Moving it is engineering, not configuration.
None of these are reasons to stay single-provider. They are the work. A multi-provider setup that ignores them is a setup that fails the first time it actually has to fail over, which is the worst possible time to discover the translation layer was never tested. The failure is quiet, too: the fallback provider returns a response, the request does not error, and only later does someone notice the output quality dropped because the prompt was never tuned for the model that served it. The rest of this post is about doing the work deliberately: failover first, then routing, then keeping prompts portable, then the limits.
How does failover routing work?
Failover routing means defining an ordered list of providers and automatically moving to the next one when the current one fails. The agent asks for “a model.” The routing layer decides which provider serves that request right now, based on which providers are currently healthy.
Three mechanisms make it work: health checks, fallback chains, and per-provider circuit breakers.
Health checks track whether each provider is currently usable. The routing layer watches every response: success rate, latency, error codes. A provider returning HTTP 500s, timing out, or rejecting requests with rate-limit errors is marked unhealthy. Health is not a static config value; it is a live measurement that changes minute to minute as providers have good and bad windows.
Fallback chains are the ordered list. Primary provider, then secondary, then tertiary. When a request to the primary fails, the routing layer retries the identical request against the next provider in the chain. The agent that made the request sees a slightly slower response, not an error. The Portkey guide on retries, fallbacks, and circuit breakers walks through the same pattern in detail.
Circuit breakers stop you from hammering a provider that is already down. Without one, every request retries the dead primary, waits for it to time out, then falls back. That is the timeout penalty paid on every single request for the entire duration of the outage. A circuit breaker watches the failure rate per provider. When failures cross a threshold, the breaker “opens” and the routing layer skips that provider entirely, going straight to the fallback. After a recovery interval the breaker goes “half-open,” lets a single probe request through, and closes again if the probe succeeds. LLM requests carry seconds of latency and real per-token cost, so failing fast on a known-bad provider saves both (n1n.ai on circuit breakers for LLM APIs).
In configuration terms, a failover-only setup is short. The chain is a list, each provider has health thresholds, and the breaker has a trip condition and a recovery interval.
providers:
anthropic:
api_key: "${ANTHROPIC_API_KEY}"
model: claude-sonnet-4-6
openai:
api_key: "${OPENAI_API_KEY}"
model: gpt-5.4
google:
api_key: "${GEMINI_API_KEY}"
model: gemini-2.5-pro
failover:
chain:
- anthropic
- openai
- google
health:
timeout: 20s
error_codes: [429, 500, 502, 503, 529]
circuit_breaker:
failure_threshold: 5
window: 30s
open_duration: 60s
half_open_probes: 1Five failures inside a 30-second window trips the breaker for that provider. The routing layer skips it for 60 seconds, then lets one probe through. The chain ordering decides who serves the traffic in the meantime. Notice the chain crosses three vendors on three different cloud footprints. If your primary and your fallback both run on the same underlying cloud region, a region outage takes out both, and you have a fallback chain that fails over to nothing.
Three details separate a failover setup that works from one that only looks like it works.
The first is error classification. Not every failure should trigger failover. A 500 or a 503 from the provider is an infrastructure failure, and failing over is correct. A 429 rate-limit response is a capacity signal, and failing over is also correct, because the provider is telling you it cannot take the request right now. But a 400 bad-request error is your fault, not the provider’s, and failing over just sends the same malformed request to the next provider, which also rejects it. The chain burns three provider calls on a request that was never going to succeed. The error_codes list in the config exists to make this distinction explicit: only the codes that mean “this provider, right now” should count toward the breaker.
The second is idempotency. Failover retries a request. If the original request was non-deterministic or had side effects, the retry is not a clean redo. For pure text generation this is rarely a problem. For agent requests that invoke tools, a retried request can invoke a tool twice. The routing layer should retry only the model call, not the tool execution, and the agent design should assume a model call can be issued more than once.
The third is the timeout budget. A fallback chain of three providers, each with a 20-second timeout, can take 60 seconds in the worst case where the first two both time out before the third answers. For a background job that is fine. For a user-facing request it is not. The fix is a total request deadline that caps the whole chain, not just each hop: if the deadline is 25 seconds, the chain stops trying after 25 seconds and returns a degraded response rather than letting the user wait a full minute. Circuit breakers help here too, because an open breaker is skipped instantly and never spends its hop’s worth of the budget.
How does cost routing work?
Failover answers “what do we do when a provider is down.” Cost routing answers a different question: “which provider should serve this request when everyone is up.” On a calm day, that is where the money is.
The principle is the one from cutting AI API costs, extended across vendors: inspect each request before it goes out, and send it to the cheapest provider and model that can actually do the job. There are three signals worth routing on.
Route by price. Sort the eligible providers by cost and pick the cheapest one that meets the quality bar for the task class. A two-sentence classification does not need a frontier model from any vendor. It needs Gemini Flash-Lite or an open model. The savings compound because the cheap tier is genuinely cheap: Flash-Lite input pricing near $0.10 per million tokens is roughly an order of magnitude below frontier input pricing, and most production traffic is not frontier-grade work.
Route by capability tier. Map task classes to tiers, not to specific models. Tier one is high-volume simple work: classification, extraction, short summaries. Tier two is moderate analysis and formatting. Tier three is genuine reasoning, long-context analysis, code generation. Each tier has a primary provider and a fallback provider, so capability routing and failover routing share the same chains. The agent sends a task class; the routing layer resolves it to a provider.
Route by latency. Some requests are user-facing and latency-sensitive. Some are background jobs where a few seconds do not matter. For the latency-sensitive path, route to whichever healthy provider currently has the lowest measured response time, even if it is not the absolute cheapest. For the background path, route purely on price. The vLLM Semantic Router project reports that routing on query complexity cut latency and token usage by roughly 47% and 48% in its benchmarks, and that semantic routing with cost-aware fallbacks can take 30 to 40% off infrastructure cost (Red Hat on vLLM Semantic Router).
The three signals are not mutually exclusive. Real routing config layers them: capability tier sets the candidate pool, then price or latency picks within it.
In config, cost routing reads as task-class rules with a candidate pool and a selection strategy.
routing:
task_classes:
classification:
tier: 1
candidates: [google-flash-lite, deepseek]
strategy: cheapest_healthy
chat_reply:
tier: 2
candidates: [openai, anthropic, google]
strategy: lowest_latency
code_review:
tier: 3
candidates: [anthropic, openai]
strategy: capability_first
batch_report:
tier: 2
candidates: [google, deepseek, openai]
strategy: cheapest_healthyThe strategy field is the lever. cheapest_healthy sorts the candidate pool by price and skips any provider whose circuit breaker is open. lowest_latency sorts by measured response time. capability_first keeps the pool in the order you ranked it by quality. Every strategy respects circuit-breaker state, so routing and failover never fight each other: an unhealthy provider is invisible to the router no matter which strategy is active.
The savings are not abstract. Consider a workload of 100,000 requests per month, split the way most production traffic actually splits: 60% tier-one classification and extraction, 30% tier-two analysis, 10% tier-three reasoning, each averaging 1,000 input and 400 output tokens.
Send all of it to a single frontier model at $3.00 input and $15.00 output, and the input cost is 100,000 times 1,000 tokens at $3.00 per million, which is $300, plus output of 100,000 times 400 tokens at $15.00 per million, which is $600. The bill is $900 per month, and the 60,000 classification requests are paying frontier rates to do work a cheap model does just as well.
Now route by tier. The 60,000 tier-one requests go to Gemini Flash-Lite at $0.10 input and $0.40 output: $6 input plus $9.60 output, about $16. The 30,000 tier-two requests go to Gemini 2.5 Pro at $1.25 and $10.00: about $37.50 input plus $120 output, around $158. The 10,000 tier-three requests stay on the frontier model: $30 input plus $60 output, $90. The total lands near $264 per month against $900. The same 100,000 requests, the same outputs, roughly a 70% reduction, and the only change is that each request now pays for the capability it actually needs. This is the multi-vendor version of the result from cutting AI API costs without code changes, and the cross-vendor spread makes the gap wider than single-provider routing can reach, because no single provider offers a tier-one option as cheap as the cheapest provider’s.
One honest caveat on these numbers: they assume your tier-one classifier sends tier-one work. Misclassification is the failure mode. A request that needs reasoning but gets routed to Flash-Lite produces a worse answer, and if that answer reaches a user, the saved cents are not worth it. Routing quality is a real engineering surface, which is why the incremental adoption section below routes the lowest-risk traffic first and measures before expanding.
How do you keep prompts portable?
Routing and failover both assume the same request can go to any provider in the chain. That assumption is false by default. Anthropic, OpenAI, and Google speak three different API dialects, and a prompt tuned for one model is not tuned for another. Portability is the layer that makes the assumption true. It is also where multi-provider gets genuinely hard.
The mechanism is an abstraction layer with a normalized interface. Your agent code writes one request format. The abstraction layer translates that format into whichever provider’s dialect the routing layer selected: it maps the system prompt into the right field, restructures the message history, rewrites tool definitions into the target schema, and parses the provider-specific response back into one normalized shape. The agent never sees the dialect. It writes once, and the layer makes it run anywhere. The major LLM gateways exist precisely to be this layer: LiteLLM exposes 100-plus models behind one OpenAI-compatible API, OpenRouter fronts 200-plus models behind one key, and Portkey offers the same as a managed gateway (Portkey vs LiteLLM vs OpenRouter comparison).
You do not have to build the abstraction layer. The build-versus-buy decision is mostly settled in favor of buy, because the translation logic, the health checks, and the circuit-breaker state machine are solved problems. The three common options trade off differently:
| Tool | Model | Routing control | Cost model | Best fit |
|---|---|---|---|---|
| LiteLLM | Open-source, self-hosted | Full YAML control over chains and strategies | No markup, you run the infrastructure | Teams wanting control and zero per-request fees |
| OpenRouter | SaaS marketplace | Preset routing variants by price, throughput, quality | Markup on token spend | Fast start, no infrastructure to run |
| Portkey | Managed gateway, self-host option | Routing plus caching and observability built in | Per-log or managed pricing | Teams wanting an integrated production stack |
OpenRouter’s markup is the clearest tradeoff to weigh: a markup on token spend is small at low volume and becomes a real line item at high volume, where a self-hosted LiteLLM with no per-request fee pays for the infrastructure it costs (PocketLantern gateway brief). Building your own is justified only when you have a compliance, air-gap, or routing-logic requirement that none of the three meets, which is rare.
This is the leaky-abstraction problem, and it is worth naming honestly rather than pretending the layer is perfect. An abstraction layer can normalize the shape that all providers share: messages, a system prompt, tool calls, a text response. It cannot normalize what only one provider has. Anthropic’s explicit cache breakpoints have no equivalent in OpenAI’s fully automatic caching. Reasoning-effort controls differ by vendor. Vision input formats, structured-output guarantees, and context-window sizes are all uneven. When you write to the normalized interface, you are writing to the lowest common denominator of every provider in your chain. Any feature that is not common is either unavailable through the abstraction or available only through a provider-specific escape hatch that breaks portability for that one call.
There is a second portability problem the abstraction layer does not touch: the prompt itself. The layer translates the API format, so the same JSON shape reaches every provider. It does not translate behavior. A system prompt tuned through a dozen iterations to make GPT produce clean structured output is calibrated to GPT, and Claude or Gemini may read the same words and behave differently. Few-shot examples that anchor one model can mislead another. The abstraction layer makes the request runnable everywhere; it does not make the request good everywhere.
The practical handling is a prompt set per task class that has been validated on every provider in that class’s chain, not a single prompt assumed to be universal. For most prompts the differences are small and one prompt works acceptably across providers. For the prompts where output quality is load-bearing, you keep provider-specific variants and the abstraction layer selects the variant matching the provider it routed to. That is more maintenance, and it is the honest price of a fallback that actually works rather than one that merely runs.
That is the real tradeoff, stated plainly. The abstraction gives you portability and costs you access to each provider’s best individual features, plus the work of keeping prompts good rather than merely runnable across providers. Most teams accept it because lowest-common-denominator capability that fails over cleanly beats a single provider’s premium capability that does not fail over at all. But it is a choice, not a free lunch, and you should make it knowing what the layer cannot do. An abstraction layer that promises full portability is overselling. One that promises a normalized shape and is honest about the leaks is telling you the truth.
What are the limits of multi-provider?
Multi-provider buys resilience, cost control, capability coverage, and leverage. It is not magic, and a post that only sold the upside would be dishonest. Here is where it stops.
You lose the best of every provider. This is the leaky-abstraction tradeoff again, and it deserves to be counted as a cost, not a footnote. Writing to a normalized interface means writing to the feature set every provider shares. The vendor-specific capabilities you would reach for in a single-provider build are either gone or behind an escape hatch that breaks portability. If a provider-exclusive feature is core to your product, multi-provider asks you to give it up or special-case it.
The testing burden multiplies. A single-provider product tests against one model’s behavior. A multi-provider product has to validate that every prompt produces acceptable output on every provider in its fallback chain, because failover means any of them might actually serve a real request. Three providers in a chain is roughly three times the prompt-evaluation surface. Skip this work and your failover is theoretical: it routes to a provider you never confirmed can do the job, and you find out during the outage.
Cache fragmentation. Provider-side prompt caching is per-provider. As prompt caching vs semantic caching explains, Anthropic’s cache lives on Anthropic’s servers and OpenAI’s lives on OpenAI’s. The moment you route the same logical request to different providers, you fragment the prompt cache: a prefix cached on Anthropic does nothing for the next request that routes to Google. Multi-provider routing and provider-side prompt caching work against each other. The honest fix is to put a provider-agnostic semantic cache at the routing layer so cache hits survive routing, but that is another component to run.
Operational surface grows. More providers means more API keys to rotate, more rate limits to track, more billing accounts to reconcile, more status pages to watch, more SDKs or one gateway to keep current. The resilience is real and so is the overhead. A two-person team should weigh whether failover-only, with no cost routing, is enough for now.
None of this argues against multi-provider. It argues for being deliberate. The honest claim is not that multi-provider solves reliability and cost. It is that multi-provider is the only way to stop a single vendor from being a single point of failure and a single non-negotiable price, and that it costs you peak per-provider capability, a multiplied testing surface, and real operational work in exchange.
How do you adopt multi-provider incrementally?
The mistake is treating multi-provider as one large migration: pick a gateway, wire in four providers, build cost routing, ship it all at once. That front-loads the hardest work, the cross-provider testing, before you have any resilience to show for it. Do it in two stages instead.
Stage one: failover only. Add exactly one second provider. Build the fallback chain and the per-provider circuit breaker. Do not build cost routing yet. Every request still goes to your existing primary at the existing price; the only change is that an outage now fails over instead of failing. This stage is small, the configuration is short, and it delivers the headline benefit, resilience, on its own. Run it in production for a few weeks. Watch how often the secondary actually gets used. Watch whether your prompts produce acceptable output when they do route to the secondary, because that is the cross-provider testing burden showing up in real traffic where you can measure it instead of guess at it.
Stage two: add routing. Once failover is proven and you trust the secondary provider on real requests, layer cost routing on top. Start with the highest-volume, lowest-risk task class, the tier-one classification and extraction work, and route only that to a cheaper provider. Measure the cost delta and the output quality for two weeks before expanding. Then add the next task class. The candidate pools in your routing config are the same chains you already built for failover, so routing is an extension of stage one, not a rebuild.
This order matters because it front-loads the benefit and back-loads the risk. Stage one is low-risk and pays off immediately. Stage two is where the testing burden and the leaky-abstraction tradeoffs live, and by the time you reach it you have weeks of real failover data telling you which providers you can actually trust. The team that does cost routing first, before proving failover, is optimizing the price of a system it has not yet made reliable.
FAQ
Do I need multi-provider if my single provider has 99.9% uptime?
99.9% uptime is still about 8.7 hours of downtime per year, and providers more commonly publish numbers around 99%, which is more than three and a half days. The Cloudflare outage on November 18, 2025 and OpenAI’s roughly 9-hour outage on December 26, 2024 both show that the downtime is not evenly sprinkled in harmless one-second increments; it arrives in multi-hour blocks. If your business can absorb a 5-hour outage landing in the middle of a workday, single-provider is a defensible choice. If it cannot, the uptime percentage is not the number that matters, the worst single outage is.
Will multi-provider make my AI feature slower?
Slightly, in two specific places. Failover adds latency only when the primary actually fails: the request attempts the primary, times out or errors, then retries the fallback, so a failed-over request is slower than a normal one. A circuit breaker minimizes this by skipping a known-bad provider instantly instead of timing out on every request. The abstraction layer itself adds a small, fixed translation cost on every request, typically negligible next to model inference time. On a healthy day with a warm circuit, multi-provider latency is effectively the latency of whichever provider you routed to.
Should I build the routing layer myself or use a gateway?
For most teams, use a gateway. The translation between provider dialects, the health checks, the circuit-breaker state machine, and the fallback logic are all solved problems, and rebuilding them is rarely justified. LiteLLM is the common self-hosted choice with full YAML control over routing. OpenRouter is the zero-setup SaaS option, with the tradeoff of a markup on spend. Portkey is the managed enterprise gateway. Building your own is worth it only when you have specific compliance, air-gap, or routing-logic requirements that no gateway meets.
Does multi-provider break prompt caching?
It fragments provider-side prompt caching, yes. Each provider maintains its own cache, so routing the same logical request across providers means a prefix cached on one provider does nothing on another. The way to keep caching benefits under multi-provider routing is a provider-agnostic semantic cache at the routing layer, which stores complete responses and serves them regardless of which provider generated them. The two caching approaches and how they compose are covered in prompt caching vs semantic caching.
How is failover routing different from a policy engine?
They operate at the same layer and answer different questions. Failover routing decides which provider serves a request based on provider health. A policy engine decides whether a request should be allowed at all, and enforces tool allowlists, budget caps, and audit logging, as covered in what is an AI agent policy engine. A complete proxy does both: it governs the request, then routes it. Routing without governance still lets an agent run up an unbounded bill across every provider in the chain; governance without routing still has a single point of failure. They compose.
Disclosure: Govyn is an open-source AI governance proxy. It includes multi-provider routing, failover chains, and circuit breakers alongside its policy engine, so we have a commercial interest in proxy-layer routing. The outage history, provider pricing, and routing tools cited here are drawn from published incident reports, provider status pages, and public documentation, all linked inline. The architectural case for multi-provider does not depend on which proxy you run. Evaluate the evidence independently.
Govyn is open source, MIT licensed. Self-host or cloud-hosted. Multi-provider routing and failover ship in core.