Skip to content

Guidance

Guidance programs mix fixed text, generation and structure. Point its OpenAI model at the gateway and every call the program makes is checked against your policies, charged to the right team and recorded in Request logs, and the same program can run on Claude or Gemini as well as GPT.

What it is

Guidance is a Python library for steering language models. With a local model it constrains generation token by token; with a remote model it sends Chat Completions requests through the openai package. guidance.models.OpenAI is the remote model to use with the gateway.

The two settings

Setting models.OpenAI argument Value
Base address base_url (passed on to the openai client; OPENAI_BASE_URL works too) https://<your-gateway>/v1
Key api_key (or OPENAI_API_KEY) Your Vidai key (starts vk_), never a provider's key

The gateway's address is under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.

A minimal call

Remote models need chat blocks (system(), user(), assistant()):

from guidance import models, gen, user, assistant

lm = models.OpenAI(
    "gpt-4o-mini",
    base_url="https://<your-gateway>/v1",
    api_key="vk_…",
)

with user():
    lm += "Name one Dutch city."
with assistant():
    lm += gen("city", max_tokens=10)

print(lm["city"])

For Claude or Gemini, put a name such as claude-sonnet-4.5 or gemini-3.5-flash in the first argument and keep the same base_url. The gateway translates the request and the answer.

Streaming

Guidance always streams from remote models, and asks for the usage chunk. A stream from Claude or Gemini arrives as OpenAI chunks; Vidai records the tokens and the cost either way.

Tool calls

Tools you pass to gen(tools=[…]) travel as OpenAI tool calls and work on every provider the gateway translates to. Each round trip is its own call in Request logs.

JSON output

guidance.json with a schema (a dict or a Pydantic model) sends response_format with a strict JSON schema. The gateway carries it to Claude and Gemini models in their own form, so the schema holds on every provider.

from guidance import json as gen_json
from pydantic import BaseModel

class Shop(BaseModel):
    city: str
    staff: int

with assistant():
    lm += gen_json(name="shop", schema=Shop)
print(lm["shop"])

Reading what the gateway did

Guidance doesn't show response headers. Look the call up in Request logs: filter on Key, Model or Path (/v1/chat/completions). Cost is never in the answer.

Errors

Gateway errors come back as the openai package's exceptions (openai.AuthenticationError, openai.PermissionDeniedError, openai.RateLimitError and so on); e.response.headers["x-vidai-error-code"] gives the stable code, such as invalid_api_key, model_not_allowed or content_guardrail_blocked. The full list is in Headers and errors.

Errors such as "not yet supported for OpenAI" or "require chat blocks" come from Guidance itself, before anything is sent: see the table below.

Model names

The first argument is a name your organisation approved under Models in the Control room. GET https://<your-gateway>/v1/models with your key lists exactly the models it may call.

What goes through the gateway

Feature Through the gateway
gen, json, tools, images (with gpt-4o* names) Yes, to OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models
select, gen(regex=…), gen(stop=…), grammars No: Guidance refuses these for remote models before sending anything; it constrains token by token only with a local model
models.Anthropic, models.Gemini Not maintained in current Guidance: use models.OpenAI with the gateway for every provider

Check it works in a minute

python -c "
from guidance import models, gen, user, assistant
lm = models.OpenAI('gpt-4o-mini', base_url='https://<your-gateway>/v1', api_key='vk_…', echo=False)
with user(): lm += 'Reply with exactly: ok'
with assistant(): lm += gen('r', max_tokens=5)
print(lm['r'])"

A short answer means the address and key are right. The call shows in Request logs within seconds.

Good to know

  • Guidance sends max_completion_tokens for max_tokens; the gateway maps it for every provider.
  • Image input is switched on by model name (names starting gpt-4o or o1). An alias with another name sends text only.
  • The openai client underneath retries twice by default; pass max_retries=0 to models.OpenAI (it goes to the client) if you don't want its retries on top of the gateway's fallback.
  • Guidance's own docs: README.