Skip to content

Streaming

Streaming works as your SDK documents it, in every format and to every provider. What's worth knowing:

Topic Behaviour
Your own events A translated stream arrives in your SDK's own event protocol. An Anthropic client gets message_start, content_block_start, content_block_delta, content_block_stop, message_delta and message_stop; a Gemini client gets Gemini chunks; an OpenAI client gets chat.completion.chunk events
Usage An OpenAI client gets the usage chunk only when it asks (stream_options: {"include_usage": true}). The gateway asks the provider for usage either way, so every streamed call is priced in Request logs
Headers The response headers, x-vidai-model and x-vidai-request-id included, arrive before the first event
Personal data put back When a guardrail pseudonymised a prompt, the originals are restored in the stream too. To do that, the gateway holds back the last few characters of each chunk until the next one shows whether a stand-in continues; the text is the same, the chunk boundaries may differ, and x-vidai-pseudonymised: 1 says so
Reasoning Reasoning from the model reaches an Anthropic or Gemini client only when that client asked for it (Anthropic thinking, Gemini thinkingConfig.includeThoughts)
Trying the next model Happens only before anything has reached you, never mid-stream. It triggers on a connection error, a time-out, 429 or 5xx; at most 3 attempts in all
An error mid-stream Once the stream has started, a provider failure closes the connection. The call is recorded in Request logs as upstream_closed
Cost of translation 1 to 2 ms per call, and a little per chunk

Read a stream with its headers

with client.chat.completions.with_streaming_response.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a haiku about canals."}],
    stream=True,
) as response:
    print(response.headers.get("x-vidai-request-id"))
    for chunk in response.parse():
        if chunk.choices and chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="")
const { data: stream, response } = await client.chat.completions
  .create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "Write a haiku about canals." }],
    stream: true,
  })
  .withResponse();
console.log(response.headers.get("x-vidai-request-id"));
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
curl -N -sS -D /dev/stderr "$VIDAI_GATEWAY/v1/chat/completions" \
  -H "Authorization: Bearer $VIDAI_KEY" -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o-mini", "stream": true,
       "messages": [{"role": "user", "content": "Write a haiku about canals."}]}'

Good to know

  • If your own server sits between your app and the gateway, give it long enough read and send time-outs for long streams, and turn off response buffering.
  • A stream that ends with a closed connection is a failed call: retry it as a whole, not from where it stopped.