Skip to content

Rate limits

One agent stuck in a loop can use up a provider's whole quota and take every other app down with it. Vidai puts a limit on each key, in requests a minute, enforced by the gateway on every call. A runaway caller is held back on its own key; everyone else carries on.

How it works

  • Each key may make a number of requests a minute. A key with no limit of its own follows the default for its kind of key: one default for people's keys and one for agents' keys, set by an organisation admin. With no default and no limit of its own, a key has No limit.
  • A caller over its limit is refused by the gateway until the minute passes. The refusal comes in the caller's own SDK format with HTTP 429 and x-vidai-error-code: rate_limit_exceeded.
  • Changing a limit or a default reaches the gateway within seconds.
  • The gateway counts requests, not tokens. For money, use a budget.
flowchart LR
    C["Calls this minute"] --> Q{"Under the key's limit?"}
    Q -->|yes| GW["Gateway carries on:<br/>model, rules, provider"]
    Q -->|no| R["429 rate_limit_exceeded<br/>until the minute passes"]

Who may set what

A key's holder, or an agent's owner, may give a key a lower limit than the default. Only an admin over the key (an organisation admin, or a group admin of its team) may set a higher one.

What you see

Each key says which limit applies: "60 requests a minute", "600 requests a minute (the default)" or "No limit". A key with a limit shows This minute: how many calls it has made this minute against its limit, as a meter, with Near its limit from 80% and At its limit when it's reached. It's a reading, refreshed every few seconds, not a record.

A key's This minute meter: calls made this minute against its limit, near its limit

Several gateway nodes

With several gateway nodes counting together, the nodes share their counts every few seconds (every second unless an organisation admin changes it in Settings › Gateway), so a key's limit holds across all of them; a key can go over by about that much of a minute's calls. Status shows how rate limits are counted on your installation: by every node together, or by each node on its own.