Skip to content

Performance

Purpose: governing every call is only acceptable if it doesn't slow your app down. The gateway applies keys, routing, Guardrails and recording in the path of each call, and adds well under a millisecond of its own.

The number

47,200 calls a second at 0.58 ms median.

Measured under these conditions:

  • the gateway's 0.9.0-beta.5 build, on one machine;
  • 32 concurrent callers;
  • every call recorded, to the same database the Control room reads;
  • a mock provider, so the time is the gateway's own, not a model's.

It's the gateway team's own run, not a published benchmark.

What it means for your app

  • A model's answer takes hundreds of milliseconds to seconds. Half a millisecond of gateway time is lost in that.
  • Translation between SDK formats adds 1 to 2 ms to a call, and a little per chunk while streaming (Any SDK to any provider).
  • Trying the next model in a fallback chain costs nothing on a call that succeeds.
  • Guardrails add the time their checks take. Pattern checks are quick; VidaiGuard checks run a model of their own.

Measure it yourself

Time the same call with and without the gateway, from the same machine, a few hundred times, and compare the medians. Use a small model and a short prompt, so the provider's own variation doesn't hide the difference.

for i in $(seq 1 200); do
  curl -s -o /dev/null -w '%{time_total}\n' "$VIDAI_GATEWAY/v1/chat/completions" \
    -H "Authorization: Bearer $VIDAI_KEY" -H "Content-Type: application/json" \
    -d '{"model": "gpt-4o-mini", "max_tokens": 1, "messages": [{"role": "user", "content": "Hi"}]}'
done | sort -n | awk '{a[NR]=$1} END {print "median", a[int(NR/2)+1], "s"}'

Request logs shows each call's time too: sort by slowest first to find the outliers.