Skip to content

Batches

Large offline jobs (classifying a backlog, summarising an archive) are cheaper as a batch, and just as much in need of governance. Through the gateway, a batch goes to the provider as a batch, at the provider's batch price, and the models it names are checked against your key before it starts.

What it is

OpenAI's Batch API (/v1/batches: create, retrieve, list, cancel). You upload a JSON Lines file of requests, create a batch that refers to it, and download the results file when it's done. The gateway passes these calls on unchanged to OpenAI or Azure OpenAI, to the same provider that holds your files: see which provider holds your files.

The two settings

The same as for the OpenAI SDK: base address https://<your-gateway>/v1 and your Vidai key. See OpenAI SDK for Python.

A minimal call

Each line of requests.jsonl is one request:

{"custom_id": "row-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Summarise: …"}]}}
from openai import OpenAI

client = OpenAI()

batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

batch = client.batches.retrieve(batch.id)
if batch.status == "completed":
    print(client.files.content(batch.output_file_id).text)
import fs from "node:fs";
import OpenAI from "openai";

const client = new OpenAI();

const batchFile = await client.files.create({ file: fs.createReadStream("requests.jsonl"), purpose: "batch" });
let batch = await client.batches.create({
  input_file_id: batchFile.id,
  endpoint: "/v1/chat/completions",
  completion_window: "24h",
});

batch = await client.batches.retrieve(batch.id);
if (batch.status === "completed" && batch.output_file_id) {
  console.log(await (await client.files.content(batch.output_file_id)).text());
}
curl https://<your-gateway>/v1/batches \
  -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"input_file_id": "file-…", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'

Upload the input file through the gateway, not directly at the provider: then the gateway knows which models it names.

Which models a batch may use

If your key is limited to some models, the gateway checks every model the input file names when you create the batch, as it would a chat call:

  • A file uploaded through the gateway: the models were noted as it uploaded. A row with a model the key can't use, or one a routing rule would deny or redirect, refuses the whole batch with 403 model_not_allowed, naming the model and how many rows use it.
  • A file the gateway didn't see: it's fetched from the provider and checked. If it can't be read, or isn't JSON Lines with a model on each row, the batch gets 403 batch_file_unverified.

A key with no model limits isn't checked.

Reading what the gateway did

Every call carries x-vidai-request-id (paste it into Request logs › Request reference). Request logs show the batch's own calls: creating it, checking on it and downloading its results. The rows run at the provider, not through the gateway, so their usage is on the provider's bill.

Errors

Status Code When
403 model_not_allowed The input file names a model your key may not use
403 batch_file_unverified Your key is limited to some models and the input file couldn't be checked
400 bad_request Several providers could serve files and none is chosen
400 provider_override_not_allowed x-provider names a provider that serves none of your key's models

The other codes are those of Chat Completions.

What goes through the gateway

Feature Through the gateway
OpenAI batches: create, retrieve, list, cancel Yes, to the provider that holds your files
Batches of Claude or Gemini requests No: batches run at OpenAI or Azure OpenAI only
Anthropic message batches (/v1/messages/batches) No

Check it works in a minute

curl https://<your-gateway>/v1/batches -H "Authorization: Bearer $OPENAI_API_KEY"

A list (perhaps empty) means the address and key are right.

Good to know

  • Guardrails don't read the requests inside a batch file.
  • The gateway never tries a fallback for batches.