Skip to content

Routing and fallback

Moving an application to a cheaper model, trialling a new one, or surviving a provider outage usually means a code change in every app, and a retry loop in each. In Vidai, routing decides where an allowed call goes, in the request path: send one model to another, split traffic between two to compare them, or fall back to the next model when a provider fails. Programs keep calling the same name; you see what each rule saved.

What it is

Routing comes after access. What a key may call is decided first, by its model access (and, on Enterprise, region rules). Routing never lets a key use a model it couldn't, and never blocks one it could. A target is a model on a provider, in any request format: the gateway translates between formats both ways.

There are five kinds, each set at a level and reaching the keys below it:

Kind What it does
Routing rule Calls to one model go to another: "Calls to gpt-4o from Team: Finance go to gpt-4o-mini"
A/B experiment Calls to one model are split between two arms by a share you set, with results per arm
Fallback chain When a model's provider is failing, the chain's models are tried in the order you set
Compliance route Enterprise A model always goes to an approved deployment (the EU one, say), with no exemptions
Region rule Enterprise Calls from a level stay within an area. See Regions and data residency

What Enterprise includes depends on the features your licence includes. On Community, routing rules, A/B experiments and fallback chains are all there.

Which rule wins

When several rules reach the same call, one order decides, and you never type a priority:

flowchart TD
  A[1 · Compliance route] -->|none| B[2 · Required rule<br/>unless the key is exempt]
  B -->|none| C[3 · A/B experiment]
  C -->|none| D[4 · Default rule<br/>closest to the key wins]
  D -->|none| E[The model asked for]
  • A Default rule lets a lower level set its own; the one closest to the key wins. The organisation can send gpt-5 to gpt-5-mini while Finance sends it to an Azure EU deployment.
  • A Required rule beats every rule below it. One key is let off only by a named exemption, with a reason and an optional end.

The Which rule wins tab shows this ladder, and Explain a key (also on every key's page) shows for each model which rule decides that key's calls, or that they're refused. Each other rule says why it doesn't apply: "A required rule above it applies", "This key is exempt", "Paused".

Models & routing › Routing › Which rule wins: one key explained, with the deciding rule marked for each model

Fallback when a provider fails

A fallback chain protects one model. When the gateway sees its provider failing, it tries the chain's models in order before the caller sees anything. Two choices shape it:

  • When every model in the chain fails: Try the model asked for anyway (the default), or Refuse the call, so the program gets a clear "service unavailable" and can try again later.
  • When the model called fails: Return its error (the default), or Try the next model, up to 3 attempts in all. A provider may charge for an attempt that failed after it received the call.

The answer's headers say what was tried, and the call's page in Request logs lists each model the chain called, Answered or Failed with why.

A fallback chain drawn as a flow: Asked for, Fallback 1, Fallback 2, with what happens if every one fails

Everything routed is measured

  • Saved by routing in the last 30 days sits under the page title, one figure per currency, with extra cost shown where a target costs more.
  • Before you save a rule, Check what it changes gives an Estimate from recent calls: what the keys it decides sent to that model a day over the last 7 days, and what that would cost at the new model's price.
  • An A/B experiment's page shows calls, failures, typical and slowest latency, tokens and cost per arm.
  • A rule says what it really does: Acts for 9 of 12 keys, Not acting here (a stronger rule decides), Already in place, Paused. When a target can no longer be used (switched off, access narrowed, off the approved list), the rule pauses itself and says why; Home tells you.

Models & routing › Routing: rules read as sentences, grouped by level, with state, purpose and saving