Skip to content

Read provider health and circuits

Purpose: providers go down. When calls to a model keep failing, the gateway's circuit breaker stops sending it calls for a while and starts its fallback chains, so your apps get answers from the next model instead of errors. This page shows you how to see that happening, tune it, and bring a provider back by hand.

Before you start

  • You're an organisation admin (organisation viewers and Compliance officers Enterprise read health; only admins tune or reset).

Steps: see what's down

  1. Open System › Status and read Providers: the providers with a circuit open or recovering, each linking to its page, with how many of its models are affected. Every provider is taking calls when none is.
  2. Open a provider from there (or from Models & routing › Providers). On its Overview, Health at the gateway shows its state, Taking calls, Recovering or Refusing calls, and each model's circuit: failures out of calls, when it tries again, and since when.

Health at the gateway for azure-openai-francecentral: its model's circuit open, failures out of calls, and when it tries again

Status also shows Trying the next model: calls the gateway sent to the next model in a chain since it started, and the limits it keeps to.

Steps: bring a provider back by hand

When the provider has recovered and you don't want to wait for the pause to end:

  1. On the provider's Overview, choose Reset circuits….
  2. Confirm. The gateway sends calls to its models again at once. If the provider is still failing, the circuits open again after a few failed calls.

Steps: tune when the gateway gives up on a provider

  1. Open Models & routing › Providers and choose Circuit breaker. It applies to every provider, and says whether these are the gateway's own settings or your organisation's.
  2. Set:
  3. Failures that stop calls: how many failed calls within the window stop calls to a provider;
  4. Window, in seconds: the time failures are counted in;
  5. Fewest calls before it can stop them: so one failure on a quiet model doesn't trip it;
  6. Pause, in seconds: how long calls stay stopped before the gateway tries again (at least 30);
  7. Answers that count as failures: status codes, for example 429, 500, 502, 503, 504.
  8. Save. "Saved: the gateway uses these now."

The Circuit breaker settings: failures, window, fewest calls, pause and the answers that count as failures

Check it worked

The provider's Health at the gateway reads Taking calls, and it's gone from Providers on Status. In Request logs, calls to its models show Answered by that provider again rather than a Fallback mark.

Good to know

  • Vidai reads health from the gateway every 15 seconds.
  • Each gateway node keeps its own circuits; the one shown is the node Vidai reaches.
  • An open circuit refuses calls to that model unless a fallback chain covers it.