Batches¶
Large offline jobs (classifying a backlog, summarising an archive) are cheaper as a batch, and just as much in need of governance. Through the gateway, a batch goes to the provider as a batch, at the provider's batch price, and the models it names are checked against your key before it starts.
What it is¶
OpenAI's Batch API (/v1/batches: create, retrieve, list, cancel). You upload a JSON Lines file of requests, create a
batch that refers to it, and download the results file when it's done. The gateway passes these calls on unchanged to
OpenAI or Azure OpenAI, to the same provider that holds your files: see
which provider holds your files.
The two settings¶
The same as for the OpenAI SDK: base address https://<your-gateway>/v1 and your Vidai key. See
OpenAI SDK for Python.
A minimal call¶
Each line of requests.jsonl is one request:
{"custom_id": "row-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Summarise: …"}]}}
from openai import OpenAI
client = OpenAI()
batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
input_file_id=batch_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
batch = client.batches.retrieve(batch.id)
if batch.status == "completed":
print(client.files.content(batch.output_file_id).text)
import fs from "node:fs";
import OpenAI from "openai";
const client = new OpenAI();
const batchFile = await client.files.create({ file: fs.createReadStream("requests.jsonl"), purpose: "batch" });
let batch = await client.batches.create({
input_file_id: batchFile.id,
endpoint: "/v1/chat/completions",
completion_window: "24h",
});
batch = await client.batches.retrieve(batch.id);
if (batch.status === "completed" && batch.output_file_id) {
console.log(await (await client.files.content(batch.output_file_id)).text());
}
curl https://<your-gateway>/v1/batches \
-H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" \
-d '{"input_file_id": "file-…", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
Upload the input file through the gateway, not directly at the provider: then the gateway knows which models it names.
Which models a batch may use¶
If your key is limited to some models, the gateway checks every model the input file names when you create the batch, as it would a chat call:
- A file uploaded through the gateway: the models were noted as it uploaded. A row with a model the key can't use, or
one a routing rule would deny or redirect, refuses the whole batch with 403
model_not_allowed, naming the model and how many rows use it. - A file the gateway didn't see: it's fetched from the provider and checked. If it can't be read, or isn't JSON Lines
with a model on each row, the batch gets 403
batch_file_unverified.
A key with no model limits isn't checked.
Reading what the gateway did¶
Every call carries x-vidai-request-id (paste it into Request logs › Request reference). Request logs show the
batch's own calls: creating it, checking on it and downloading its results. The rows run at the provider, not through
the gateway, so their usage is on the provider's bill.
Errors¶
| Status | Code | When |
|---|---|---|
| 403 | model_not_allowed |
The input file names a model your key may not use |
| 403 | batch_file_unverified |
Your key is limited to some models and the input file couldn't be checked |
| 400 | bad_request |
Several providers could serve files and none is chosen |
| 400 | provider_override_not_allowed |
x-provider names a provider that serves none of your key's models |
The other codes are those of Chat Completions.
What goes through the gateway¶
| Feature | Through the gateway |
|---|---|
| OpenAI batches: create, retrieve, list, cancel | Yes, to the provider that holds your files |
| Batches of Claude or Gemini requests | No: batches run at OpenAI or Azure OpenAI only |
Anthropic message batches (/v1/messages/batches) |
No |
Check it works in a minute¶
curl https://<your-gateway>/v1/batches -H "Authorization: Bearer $OPENAI_API_KEY"
A list (perhaps empty) means the address and key are right.
Good to know¶
- Guardrails don't read the requests inside a batch file.
- The gateway never tries a fallback for batches.