Guidance¶
Guidance programs mix fixed text, generation and structure. Point its OpenAI model at the gateway and every call the program makes is checked against your policies, charged to the right team and recorded in Request logs, and the same program can run on Claude or Gemini as well as GPT.
What it is¶
Guidance is a Python library for steering language models. With a local
model it constrains generation token by token; with a remote model it sends Chat Completions requests through the
openai package. guidance.models.OpenAI is the remote model to use with the gateway.
The two settings¶
| Setting | models.OpenAI argument |
Value |
|---|---|---|
| Base address | base_url (passed on to the openai client; OPENAI_BASE_URL works too) |
https://<your-gateway>/v1 |
| Key | api_key (or OPENAI_API_KEY) |
Your Vidai key (starts vk_), never a provider's key |
The gateway's address is under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.
A minimal call¶
Remote models need chat blocks (system(), user(), assistant()):
from guidance import models, gen, user, assistant
lm = models.OpenAI(
"gpt-4o-mini",
base_url="https://<your-gateway>/v1",
api_key="vk_…",
)
with user():
lm += "Name one Dutch city."
with assistant():
lm += gen("city", max_tokens=10)
print(lm["city"])
For Claude or Gemini, put a name such as claude-sonnet-4.5 or gemini-3.5-flash in the first argument and keep the
same base_url. The gateway translates the request and the answer.
Streaming¶
Guidance always streams from remote models, and asks for the usage chunk. A stream from Claude or Gemini arrives as OpenAI chunks; Vidai records the tokens and the cost either way.
Tool calls¶
Tools you pass to gen(tools=[…]) travel as OpenAI tool calls and work on every provider the gateway translates to.
Each round trip is its own call in Request logs.
JSON output¶
guidance.json with a schema (a dict or a Pydantic model) sends response_format with a strict JSON schema. The
gateway carries it to Claude and Gemini models in their own form, so the schema holds on every provider.
from guidance import json as gen_json
from pydantic import BaseModel
class Shop(BaseModel):
city: str
staff: int
with assistant():
lm += gen_json(name="shop", schema=Shop)
print(lm["shop"])
Reading what the gateway did¶
Guidance doesn't show response headers. Look the call up in Request logs: filter on Key, Model or Path
(/v1/chat/completions). Cost is never in the answer.
Errors¶
Gateway errors come back as the openai package's exceptions (openai.AuthenticationError,
openai.PermissionDeniedError, openai.RateLimitError and so on); e.response.headers["x-vidai-error-code"] gives the
stable code, such as invalid_api_key, model_not_allowed or content_guardrail_blocked. The full list is in
Headers and errors.
Errors such as "not yet supported for OpenAI" or "require chat blocks" come from Guidance itself, before anything is sent: see the table below.
Model names¶
The first argument is a name your organisation approved under Models in the Control room.
GET https://<your-gateway>/v1/models with your key lists exactly the models it may call.
What goes through the gateway¶
| Feature | Through the gateway |
|---|---|
gen, json, tools, images (with gpt-4o* names) |
Yes, to OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models |
select, gen(regex=…), gen(stop=…), grammars |
No: Guidance refuses these for remote models before sending anything; it constrains token by token only with a local model |
models.Anthropic, models.Gemini |
Not maintained in current Guidance: use models.OpenAI with the gateway for every provider |
Check it works in a minute¶
python -c "
from guidance import models, gen, user, assistant
lm = models.OpenAI('gpt-4o-mini', base_url='https://<your-gateway>/v1', api_key='vk_…', echo=False)
with user(): lm += 'Reply with exactly: ok'
with assistant(): lm += gen('r', max_tokens=5)
print(lm['r'])"
A short answer means the address and key are right. The call shows in Request logs within seconds.
Good to know¶
- Guidance sends
max_completion_tokensformax_tokens; the gateway maps it for every provider. - Image input is switched on by model name (names starting
gpt-4ooro1). An alias with another name sends text only. - The
openaiclient underneath retries twice by default; passmax_retries=0tomodels.OpenAI(it goes to the client) if you don't want its retries on top of the gateway's fallback. - Guidance's own docs: README.