Your AI features died at 3am — and it wasn't the model's fault
A rejected payment, a retired model, a rate limit you never saw coming. Model fallbacks keep your prompts working when providers fail — without changing a single word of your prompt.
The first time we saw it, it was Google: your project’s billing gets rejected — expired card, failed charge, credit exhausted — and Gemini doesn’t tell you that. It tells you your project has been denied access.
So you check IAM roles. Then service accounts. Then API keys. An hour later you land on the billing page and it clicks: this was never about access. It was a card that expired in a drawer somewhere.
That’s the thing about AI infrastructure: when requests start failing, it’s almost never the model’s fault. It’s the boring, brutal stuff — and it takes your feature down just the same.
The ways your requests die
Here’s what actually takes down LLM features in production, in rough order of how often we see it:
- Billing and payment failures. A card expires, a payment gets rejected, an account goes over its spend limit. The provider doesn’t warn you — requests just start returning 402s or hard rate limits. And the error messages lie: Google’s your project has been denied access is really a billing problem, not an access one. You’ll debug the wrong thing for an hour.
- Provider outages. They happen more often than the status pages admit. Usually short, always at the worst time.
- Model retirements. “GPT-4o will be deprecated on this date.” The date comes, the model 404s, and the prompt that’s been working for six months is dead. You read about it in a migration email you archived.
- Rate limits. You shipped, traffic grew, and suddenly you’re one over the limit at 2pm on a Tuesday.
- Silent failures. The request “succeeds” but the model refuses the task, or returns text where you asked for JSON. No error code. Just a useless response.
Any one of these takes a feature down. And here’s the kicker: when you get hit by one, the fix always involves touching the prompt or the code that sends it — then redeploying, then praying.
What “resilience” actually requires
The naive version of fallbacks is a try/catch that calls a different model. That doesn’t survive contact with reality, because three things are harder than they look:
You have to know what actually failed. A 429 is not a 503 is not a timeout is not a refusal. If your fallback fires on the wrong things — or doesn’t fire on the right ones — you’re either burning money on retries or shipping empty responses. You need failure classification, and you need it per-attempt.
The fallback has to be able to do the job. This is the one nobody talks about. If your prompt uses a JSON schema, the fallback model has to be able to produce that schema. Claude 3.x can’t do native JSON Schema — it does JSON via prompt hints. Some models can’t do structured output at all. A chain full of models that can’t produce your format is a chain that fails elegantly, every single time.
You have to be able to see it happen. When the fallback fires, which model answered? Why did the first one fail? If you can’t answer that from your logs, you’re flying blind — and you’ll only find out your fallbacks are broken when they’re needed most.
What we shipped
We’ve been working on making this boring-robust for a while, and it’s live now.
A fallback chain that actually fires. Configure an ordered list of models per prompt. When the primary fails, the gateway walks the chain — whether the failure is a rate limit, a provider error, a timeout, or a refusal. Each entry can be limited to specific triggers: “only fall over to this model on rate limits, use the catch-all for everything else.” (Provider enablement is a policy gate, not a fallback trigger — if the org disabled a provider, requests to it error loudly so the prompt gets fixed, rather than silently routing around the decision.)
Output compatibility, enforced. Since we track structured output capability per model in our catalog, the gateway now skips any chain candidate that can’t produce the format your prompt asks for. The dashboard flags those models while you’re building the chain, so you find out in the editor, not in production.
Refusal detection. When a model decides it won’t do the task — which is a real thing that happens, especially with content policies — the gateway can detect it and move to the next model. It’s opt-in, because we’re honest about heuristics, but when you need it, it’s there.
The fallback trail. Every attempt is logged: which model, what happened, what it returned, how long it took. In the logs you see the whole story — “gpt-4o hit a 503, haiku stepped in and handled it in 400ms” — instead of a mystery response.
Test your chain without breaking anything. In the prompt editor there’s a Test selector that runs the request against any single fallback model — or every fallback in one go, so you can compare them — without ever calling the primary. A broken chain entry shows up before it costs you a production incident.
The part we’re happiest about
None of this required changing a prompt. The same prompt, the same variables, the same schema — the fallback models just handle it. That’s the point: models and providers are exchangeable. You can retire a provider, dodge a billing disaster, or swap a quality model for a cheap one on the same prompt, and the only thing that changes is which model answers.
If you’ve ever watched a status page at 3am wondering whether to rewrite your prompt for a different provider — join the waitlist. Fallbacks are live, and honestly, we wish we’d had them months ago.