Provider keys and resilience¶
When a provider rate-limits you or goes down, every app finds out at once, each with its own retry loop. The gateway absorbs it instead: several provider keys shared out, a failing key taken out of service, a failing model rested for a while, and calls moved to the next model in a fallback chain, all before a caller sees an error.
Several keys for one provider¶
A provider has a main key and can have extra upstream keys. Each has a name of your own (such as backup) and
is shown only by its last four characters. How a key is chosen:
| Choice | What happens |
|---|---|
| The main key only | Every call uses the main key; extra keys wait |
| Each key in turn | Calls go to each switched-on key in turn |
| A key at random | Each call goes to a switched-on key chosen at random |
| By weight | Keys with a higher weight take a bigger share of calls ("10000, about 50% of calls") |
What they're for: rotating a provider key with no outage (add the new key, make it the main key, remove the old one), and splitting calls between provider projects. Extra keys don't add capacity on their own: providers set their limits per organisation, project or deployment, not per key.
When the provider rate-limits or refuses one key, the gateway takes that key out of service and the provider's
other keys carry its calls. The key reads Out of service, with why (refused, so it may need replacing, or rate
limited, which passes by itself) and when it's tried again; Home tells admins the same, with Check its keys. Every answer names the provider key used in
x-vidai-provider-key (its label, never the secret).

The circuit breaker¶
When calls to a model keep failing, the gateway stops sending it calls for a while, then lets one call through to test it: a circuit per model, opened by failures and closed by a success. While a model's circuit is open, its calls go to its fallback chain if it has one, or are refused.
stateDiagram-v2
direction LR
[*] --> Taking: calls succeed
Taking --> Resting: too many failures
Resting --> Testing: after the rest
Testing --> Taking: test call works
Testing --> Resting: test call fails (longer rest)
One setting covers every provider: how many failures stop calls, in what window, the fewest calls before it can, how long calls stay stopped (at least 30 seconds), and which answers count as failures (such as 429, 500, 502, 503 and 504). Each gateway node keeps its own circuits. An organisation admin can Reset circuits… to try a model again at once.
What you see¶
A provider's Overview shows Health at the gateway: Taking calls, Recovering or Refusing calls, and each model's circuit with its failures, when it tries again and since when. Status lists every provider with a circuit open or recovering. Event webhooks can send Models out of service to your own systems as it happens.

Fallback chains¶
A fallback chain names the models to try instead, in order. The gateway picks the next one before the call is sent,
and a chain can also try the next model when a call fails (connection errors, time-outs, 429 and 5xx), always
before anything reaches the caller, never mid-stream. The answer says what happened: x-vidai-fallback,
x-vidai-fallback-from and x-vidai-fallback-attempts, and Request logs lists every model tried. See
Routing and fallback chains.
Related¶
- Providers and models
- Routing and fallback chains
- Request logs and the call record: the models a call tried
- Integrations: events to your SIEM as models go out of service and back