LLM utility libraries¶
Libraries that validate, retry, optimise or constrain a model's answers make more calls than the code suggests: a retry after a failed validation, a reasoning step, thousands of trial runs while an optimiser compiles. Point them at the gateway and every one of those calls is checked against your policies, charged to the right team and recorded in Request logs, so the hidden calls show up in the cost.
Pick your library¶
| Library | What it does | The setting(s) to change | Page |
|---|---|---|---|
| Instructor | Pydantic objects from a model's answer, with validation retries | base_url and api_key on the OpenAI or Anthropic client it wraps |
Instructor |
| DSPy | Modules and optimisers for model programs | api_base, api_key and model_type="chat" on dspy.LM, model as openai/<name> |
DSPy |
| Guidance | Templates mixing fixed text, generation and JSON | base_url and api_key on models.OpenAI |
Guidance |
| Marvin | One-line AI functions (extract, classify, cast) and agents |
OPENAI_BASE_URL, OPENAI_API_KEY, and the model as openai-chat:<name> |
Marvin |
The values are always the same two: the gateway's address and a Vidai key (it starts vk_). Connect your app, on
the Keys page and in System › Settings › Gateway, shows the address and the base URL for each kind of SDK.
For every library¶
- Use Chat Completions. Every library here speaks the OpenAI format, and the gateway translates Chat Completions to Claude, Gemini and the other providers it serves. Some libraries now default to OpenAI's Responses API, which reaches OpenAI models only: each page says which setting keeps them on Chat Completions.
- Count the retries. A validation retry is a new call with its own cost. On top of that, a fallback chain may already have tried another model before you see an error, so keep the library's or SDK's network retries few and spaced out (one or two).
- Find the calls. None of these libraries shows the gateway's response headers. Look the calls up in Request logs (filter Key, Model or Path) within seconds of being made.
- Give heavy jobs their own key. An optimiser or a batch of extractions on its own key shows apart in Request logs and Cost, and a rate limit on that key keeps it from starving everything else.
- Pick models by the names you approved under Models in the Control room. A model the key may not call is
refused with
model_not_allowed, never sent elsewhere.