DSPy¶
A DSPy program makes several model calls per answer, and an optimiser makes thousands while it compiles. Give
dspy.LM the gateway's address and a Vidai key and every one of them is checked against your policies, charged to the
right team and recorded in Request logs, whichever provider's model answers.
What it is¶
DSPy is a Python framework for programming language models with signatures and modules
(Predict, ChainOfThought, ReAct) and tuning them with optimisers. Every module calls the language model you set
with dspy.configure(lm=…).
The two settings¶
| Setting | dspy.LM argument |
Value |
|---|---|---|
| Base address | api_base |
https://<your-gateway>/v1 |
| Key | api_key |
Your Vidai key (starts vk_), never a provider's key |
| Model | first argument, with the openai/ prefix |
openai/<model name>, for example openai/claude-sonnet-4.5 |
| API | model_type |
"chat" |
The openai/ prefix tells DSPy to speak the OpenAI format to your address; the name after it is the model name the
gateway sees. Set model_type="chat": from DSPy 3.5 the default for openai/ models is the Responses API, which the
gateway sends to OpenAI models only. Chat Completions reaches every provider.
The gateway's address is under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.
A minimal call¶
import dspy
lm = dspy.LM(
"openai/gpt-4o-mini",
api_base="https://<your-gateway>/v1",
api_key="vk_…",
model_type="chat",
)
dspy.configure(lm=lm)
qa = dspy.ChainOfThought("question -> answer")
print(qa(question="Which city is the capital of the Netherlands?").answer)
For Claude or Gemini, change only the name: dspy.LM("openai/claude-sonnet-4.5", …) or
dspy.LM("openai/gemini-3.5-flash", …). The gateway translates the request and the answer. Keep the openai/ prefix
and the same api_base; don't switch to anthropic/ or gemini/.
Streaming¶
dspy.streamify streams a program's output; it works through the gateway like any Chat Completions stream. A stream
from Claude or Gemini arrives as OpenAI chunks.
Tool calls¶
dspy.ReAct and modules that use dspy.Tool work on every provider the gateway translates to. Each reasoning step and
each tool round trip is its own call in Request logs.
def opening_hours(shop: str) -> str:
return "09:00 to 17:30"
agent = dspy.ReAct("question -> answer", tools=[opening_hours])
print(agent(question="When does the Utrecht shop open?").answer)
Reading what the gateway did¶
DSPy doesn't show response headers. Look the calls up in Request logs: filter on Key, Model or Path
(/v1/chat/completions). lm.history (or dspy.inspect_history()) shows what DSPy sent and received; cost is
never in the answer, so read it in Request logs.
Errors¶
The gateway writes its errors in the OpenAI format, and DSPy raises them as exceptions with the status and message.
Common ones while setting up: 401 invalid_api_key, 403 model_not_allowed (your key may not call that model),
404 model_not_found, 400 when the Responses API reaches a non-OpenAI model (set model_type="chat"), and 429 when
an optimiser runs into a limit on the key or team. The full list is in
Headers and errors.
Model names¶
The part after openai/ is the name your organisation approved under Models in the Control room.
GET https://<your-gateway>/v1/models with your key lists exactly the models it may call.
What goes through the gateway¶
| Feature | Through the gateway |
|---|---|
All modules, ReAct, optimisers, streaming, with model_type="chat" |
Yes, to OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models |
model_type="responses" (the default for openai/ from DSPy 3.5) |
OpenAI and Azure OpenAI models only; another provider gets a clear 400 |
dspy.Embedder with an openai/ embedding model |
OpenAI and Azure OpenAI embedding models only |
Check it works in a minute¶
python -c "
import dspy
dspy.configure(lm=dspy.LM('openai/gpt-4o-mini', api_base='https://<your-gateway>/v1', api_key='vk_…', model_type='chat', cache=False))
print(dspy.Predict('prompt -> reply')(prompt='Reply with exactly: ok').reply)"
A short answer means the address and key are right. The call shows in Request logs within seconds.
Good to know¶
- DSPy caches answers by default: a repeated call never reaches the gateway, so it has no row in Request logs. Pass
cache=Falsewhile you check the set-up. num_retriesdefaults to 3. A fallback chain may already have tried another model before you see an error (connection errors, time-outs, 429 and 5xx), so lower it to 1 or 2 to avoid multiplying attempts.- Give optimiser runs their own key: compilation calls then show apart from production calls in Request logs and Cost, and a rate limit on that key keeps a large run from starving the rest.
- DSPy's own docs: Language models.