Providers fail and rate-limit you at the worst moment¶
Several keys per provider, shared out and rested when one is rate-limited. A model that keeps failing taken out for a while. Fallback chains that try the next model before the caller sees anything. And the answer says what was tried.
The problem¶
The campaign launches at nine. At five past, the provider starts answering 429s, then 503s. Every app finds out at once, each with its own retry loop, all hammering the same provider key that everyone shares. The outage is the provider's; the incident is yours.
Why it stays unsolved¶
Resilience is usually written into each app: a retry loop here, a fallback model there, a different key in a different config file. Each app does it differently, some not at all, and none of them knows what the others are doing. Rotating a provider key means a coordinated change everywhere it's used.
How Vidai handles it¶
- Several keys for one provider. Calls are shared out between them: in turn, at random, by weight, or the main key only. A key the provider rate-limits or refuses is taken out of service and the others carry its calls. Rotating a provider key is: add the new one, make it the main key, remove the old one. No outage.
- A circuit per model. When calls to a model keep failing, the gateway stops sending it calls for a while, then tests it with one call. Calls meanwhile go to its fallback chain, if it has one.
- Fallback chains. Name the models to try instead, in order. A chain can also try the next model when a call fails, always before anything reaches the caller.
- The answer says what happened:
x-vidai-fallback,x-vidai-fallback-fromandx-vidai-fallback-attemptson the answer; every model tried on the call's page in Request logs. - You see it as it happens. A provider's page shows each model's circuit; Status lists every provider with a circuit open; events can go to your own systems as models go out of service and come back.

What changes for you¶
- An outage at the provider stays at the provider. Callers get an answer from the next model in the chain.
- No retry code in your apps, and no apps retrying on top of each other.
- Key rotations stop being events. The provider key changes in one place.
- Proof: a provider's Health at the gateway, model by model.
