Fallback and retries¶
Purpose: when a provider goes down, your app should keep answering, and it shouldn't hammer a struggling provider with retries on top of retries. The gateway's fallback chains move calls to other models while a provider is failing, and every answer says which model really answered. This page says exactly what the gateway does, so you can set your own retries to match.
What the gateway does for you, and what it doesn't¶
By default, the gateway resends nothing. A call that fails at the provider comes back to you with the provider's error, in your SDK's format. What changes that is a fallback chain: an admin names, for a model, the models to try instead, in order (Build a fallback chain). A chain acts in two ways:
| When | What happens | What your app sees |
|---|---|---|
| The model's provider is failing (the gateway has stopped sending it calls for a while: its circuit is open) | Before your call is sent, the gateway picks the first model in the chain that's healthy and that your key may call, and sends the call there | An answer from that model, with x-vidai-fallback: true and x-vidai-fallback-from |
| The model called fails on this call, and the chain is set to Try the next model | The gateway sends the call again to the next model in the chain | An answer from the model that answered, with x-vidai-fallback-attempts (attempts in all, the one that answered included) |
Without Try the next model (the chain's default is Return its error), a failed call comes back to you as it failed; the failure counts towards that model's health, and once its circuit opens, later calls go down the chain.
When the gateway tries the next model¶
Only when all of these hold:
- the model called couldn't be reached, timed out, dropped the connection, or answered 429 or 5xx;
- nothing has reached you yet: a streamed answer can move to the next model until its first event, never after;
- the call is an ordinary model call: never an upload, a file, a batch or a fine-tuning job, and never a Responses
call that continues a stored response (
previous_response_id), which belongs to its provider; - the call is within the limits: at most 3 attempts, 60 seconds for all of them, and a request body of 1 MiB or less. A larger call is sent once, as without a chain.
Never on another 4xx (a bad request, a provider's content refusal) or on the gateway's own refusals (keys, rules, limits, Guardrails): the next model would refuse it too.
Each attempt is a whole call to that model: its own provider, translation and provider key, and the same checks as any call (your key's models, routing rules, region rules). A chain never sends a call to a model your key may not call, or outside a region its rules keep it in.
When every model in the chain fails¶
- Try the next model ran out: you get the last model's own error, as that provider sent it.
- No model in the chain was healthy before the call: by default the gateway tries the model you asked for anyway, and
you get whatever its provider answers. If the chain is set to Refuse the call, you get
503
fallback_exhaustedinstead: retry later.
Which model answered¶
An alias, a routing rule or a fallback can mean another model answers than the one you named. Your code never has to guess:
| Header | When | What it tells you |
|---|---|---|
x-vidai-model |
Every answer a model produced, streams and provider errors included | The model that really answered, after aliases, routing and fallback |
x-vidai-requested-model |
Only when it differs from x-vidai-model |
The model you asked for |
x-vidai-routing-rule |
A routing rule decided where the call went | The rule's id |
x-vidai-fallback, x-vidai-fallback-from |
The first choice's provider was failing, so the chain was used before the call | true, and the provider the call would have gone to |
x-vidai-fallback-attempts |
More than one attempt was made on this call | Attempts in all |
So x-vidai-requested-model present means "something changed the model"; the other headers say what. A call where
a routing rule sent gpt-4.1-mini to an EU deployment reads:
x-vidai-requested-model: gpt-4.1-mini
x-vidai-model: gpt-4.1-mini-eu
x-vidai-routing-rule: <the rule's id>
Log x-vidai-model beside your own records if what answered matters to you (for evaluation, for example):
raw = client.chat.completions.with_raw_response.create(
model="gpt-4.1-mini",
messages=[{"role": "user", "content": "Summarise this brief"}],
)
answered_by = raw.headers.get("x-vidai-model")
asked_for = raw.headers.get("x-vidai-requested-model") or answered_by
attempts = int(raw.headers.get("x-vidai-fallback-attempts", "1"))
completion = raw.parse()
const { data: completion, response } = await client.chat.completions
.create({
model: "gpt-4.1-mini",
messages: [{ role: "user", content: "Summarise this brief" }],
})
.withResponse();
const answeredBy = response.headers.get("x-vidai-model");
const askedFor = response.headers.get("x-vidai-requested-model") ?? answeredBy;
const attempts = Number(response.headers.get("x-vidai-fallback-attempts") ?? "1");
The answer's body also names a model, but that's what the provider wrote: x-vidai-model is the name your
organisation uses.
What it costs¶
- The cost is the model that answered. The call's cost in Request logs and Cost is priced from the attempt that served it.
- A failed attempt may be charged. A provider may charge for an attempt that failed after it received the call. That's why the gateway caps attempts at 3, and why your own retries should be few.
- A fallback model may cost more or less than the one you asked for. Each entry's list price is shown beside it when an admin builds the chain.
Your own retries¶
Every SDK retries on its own (most twice, with backoff). Keep that, but keep it small:
- Keep one or two retries, spaced out. A chain set to Try the next model already made up to 3 attempts before you saw the error; three of yours on top makes nine calls to a struggling provider. If no chain covers your model, your SDK's retries are the only ones, so don't turn them off.
- Retry only what's worth retrying: connection errors, time-outs, 429, 502, 503 and 504. Never 401 or 403, and
never
content_guardrail_blockedwith the same content. The full list is in Errors and refusals. - Back off on your own clock after a 429 from the gateway. The gateway's own 429 carries no
Retry-After; a provider's own 429 passes through with its headers (Limits). - Retry a broken stream as a whole. A stream that ends with a closed connection is a failed call; there's no resuming from where it stopped (Streaming).
- A retry is a new call. The gateway reads no idempotency header: each retry is recorded and charged on its own.
Send the same
x-request-idon every try, so your retries line up in Request logs (Request ids).
Check it worked¶
- In Daily › Request logs, choose More filters, then Routing › Fallback only: the calls a fallback answered.
- Open one. Answered by says Fallback ("answered by … after … failed"), and Models tried lists each model in order, Answered or Failed with why. A call none answered reads Every fallback failed.

Related¶
- Build a fallback chain: how an admin sets one up
- Model names and aliases: the name you send and the model that answers
- Response headers and Errors
- Migrate from calling providers directly: the failover code you can delete
Good to know¶
- A chain acts on outages and failed calls, not on the content of an answer.
- A provider counts as failing after repeated 429 and 5xx answers in a short window; the gateway tries it again after a pause, and the pause grows (up to five minutes) while it keeps failing. The first failures before that reach you unless the chain is set to Try the next model.
- Health is kept per gateway, per provider and model: one failing model doesn't move calls off the provider's other models.