Control that costs you speed¶
The gateway is a single small binary that adds well under a millisecond to a call: 47,200 calls a second at a 0.58 ms median, with every call recorded (the gateway's 0.9.0-beta.5 run on one machine; conditions below).
The problem¶
Every team building with AI cares about latency: chat that feels slow is chat nobody uses, and agents make many calls per task. So when someone proposes putting governance in the path of every call, the first question is fair: how much slower does everything get?
Why it stays unsolved¶
Another hop usually means another service, written for flexibility rather than speed, with its own database lookups on every request and its own queue for logging. It adds time to every call, more under load, and becomes the thing everyone blames when a call is slow. So teams route around it, and governance with it.
How Vidai handles it¶
- A single binary in the path, not a stack. The gateway decides each call itself and records it without making the caller wait.
- Measured with recording on. In the gateway's 0.9.0-beta.5 run it handled 47,200 calls a second at a 0.58 ms median: one machine, 32 concurrent callers, every call recorded, a mock provider in place of a real one. It's our own measurement, not a published benchmark, and it's updated each release.
- Several nodes when you need them. Run more gateway nodes side by side; they share their configuration and Status shows each one and whether it's up to date.
The model's own time to answer is usually measured in hundreds of milliseconds or seconds. Next to that, the gateway's share is a rounding error.
What changes for you¶
- Nobody routes around governance to save time, because there's no time to save.
- Every call recorded, at full speed. You don't trade visibility for latency.
- Proof: every call's page in Request logs shows how long it took.
