Skip to content

Provider keys and resilience

When a provider rate-limits you or goes down, every app finds out at once, each with its own retry loop. The gateway absorbs it instead: several provider keys shared out, a failing key taken out of service, a failing model rested for a while, and calls moved to the next model in a fallback chain, all before a caller sees an error.

Several keys for one provider

A provider has a main key and can have extra upstream keys. Each has a name of your own (such as backup) and is shown only by its last four characters. How a key is chosen:

Choice What happens
The main key only Every call uses the main key; extra keys wait
Each key in turn Calls go to each switched-on key in turn
A key at random Each call goes to a switched-on key chosen at random
By weight Keys with a higher weight take a bigger share of calls ("10000, about 50% of calls")

What they're for: rotating a provider key with no outage (add the new key, make it the main key, remove the old one), and splitting calls between provider projects. Extra keys don't add capacity on their own: providers set their limits per organisation, project or deployment, not per key.

When the provider rate-limits or refuses one key, the gateway takes that key out of service and the provider's other keys carry its calls. The key reads Out of service, with why (refused, so it may need replacing, or rate limited, which passes by itself) and when it's tried again; Home tells admins the same, with Check its keys. Every answer names the provider key used in x-vidai-provider-key (its label, never the secret).

The Upstream keys of azure-openai-westeurope: its main key and an extra key named standby, with their weights and share of calls

The circuit breaker

When calls to a model keep failing, the gateway stops sending it calls for a while, then lets one call through to test it: a circuit per model, opened by failures and closed by a success. While a model's circuit is open, its calls go to its fallback chain if it has one, or are refused.

stateDiagram-v2
    direction LR
    [*] --> Taking: calls succeed
    Taking --> Resting: too many failures
    Resting --> Testing: after the rest
    Testing --> Taking: test call works
    Testing --> Resting: test call fails (longer rest)

One setting covers every provider: how many failures stop calls, in what window, the fewest calls before it can, how long calls stay stopped (at least 30 seconds), and which answers count as failures (such as 429, 500, 502, 503 and 504). Each gateway node keeps its own circuits. An organisation admin can Reset circuits… to try a model again at once.

What you see

A provider's Overview shows Health at the gateway: Taking calls, Recovering or Refusing calls, and each model's circuit with its failures, when it tries again and since when. Status lists every provider with a circuit open or recovering. Event webhooks can send Models out of service to your own systems as it happens.

A provider's Health at the gateway: each model's circuit, its failures out of calls, and when it tries again

Fallback chains

A fallback chain names the models to try instead, in order. The gateway picks the next one before the call is sent, and a chain can also try the next model when a call fails (connection errors, time-outs, 429 and 5xx), always before anything reaches the caller, never mid-stream. The answer says what happened: x-vidai-fallback, x-vidai-fallback-from and x-vidai-fallback-attempts, and Request logs lists every model tried. See Routing and fallback chains.