Streaming¶
Streaming works as your SDK documents it, in every format and to every provider. What's worth knowing:
| Topic | Behaviour |
|---|---|
| Your own events | A translated stream arrives in your SDK's own event protocol. An Anthropic client gets message_start, content_block_start, content_block_delta, content_block_stop, message_delta and message_stop; a Gemini client gets Gemini chunks; an OpenAI client gets chat.completion.chunk events |
| Usage | An OpenAI client gets the usage chunk only when it asks (stream_options: {"include_usage": true}). The gateway asks the provider for usage either way, so every streamed call is priced in Request logs |
| Headers | The response headers, x-vidai-model and x-vidai-request-id included, arrive before the first event |
| Personal data put back | When a guardrail pseudonymised a prompt, the originals are restored in the stream too. To do that, the gateway holds back the last few characters of each chunk until the next one shows whether a stand-in continues; the text is the same, the chunk boundaries may differ, and x-vidai-pseudonymised: 1 says so |
| Reasoning | Reasoning from the model reaches an Anthropic or Gemini client only when that client asked for it (Anthropic thinking, Gemini thinkingConfig.includeThoughts) |
| Trying the next model | Happens only before anything has reached you, never mid-stream. It triggers on a connection error, a time-out, 429 or 5xx; at most 3 attempts in all |
| An error mid-stream | Once the stream has started, a provider failure closes the connection. The call is recorded in Request logs as upstream_closed |
| Cost of translation | 1 to 2 ms per call, and a little per chunk |
Read a stream with its headers¶
with client.chat.completions.with_streaming_response.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Write a haiku about canals."}],
stream=True,
) as response:
print(response.headers.get("x-vidai-request-id"))
for chunk in response.parse():
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
const { data: stream, response } = await client.chat.completions
.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Write a haiku about canals." }],
stream: true,
})
.withResponse();
console.log(response.headers.get("x-vidai-request-id"));
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
curl -N -sS -D /dev/stderr "$VIDAI_GATEWAY/v1/chat/completions" \
-H "Authorization: Bearer $VIDAI_KEY" -H "Content-Type: application/json" \
-d '{"model": "gpt-4o-mini", "stream": true,
"messages": [{"role": "user", "content": "Write a haiku about canals."}]}'
Good to know¶
- If your own server sits between your app and the gateway, give it long enough read and send time-outs for long streams, and turn off response buffering.
- A stream that ends with a closed connection is a failed call: retry it as a whole, not from where it stopped.