Skip to content

DSPy

A DSPy program makes several model calls per answer, and an optimiser makes thousands while it compiles. Give dspy.LM the gateway's address and a Vidai key and every one of them is checked against your policies, charged to the right team and recorded in Request logs, whichever provider's model answers.

What it is

DSPy is a Python framework for programming language models with signatures and modules (Predict, ChainOfThought, ReAct) and tuning them with optimisers. Every module calls the language model you set with dspy.configure(lm=…).

The two settings

Setting dspy.LM argument Value
Base address api_base https://<your-gateway>/v1
Key api_key Your Vidai key (starts vk_), never a provider's key
Model first argument, with the openai/ prefix openai/<model name>, for example openai/claude-sonnet-4.5
API model_type "chat"

The openai/ prefix tells DSPy to speak the OpenAI format to your address; the name after it is the model name the gateway sees. Set model_type="chat": from DSPy 3.5 the default for openai/ models is the Responses API, which the gateway sends to OpenAI models only. Chat Completions reaches every provider.

The gateway's address is under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.

A minimal call

import dspy

lm = dspy.LM(
    "openai/gpt-4o-mini",
    api_base="https://<your-gateway>/v1",
    api_key="vk_…",
    model_type="chat",
)
dspy.configure(lm=lm)

qa = dspy.ChainOfThought("question -> answer")
print(qa(question="Which city is the capital of the Netherlands?").answer)

For Claude or Gemini, change only the name: dspy.LM("openai/claude-sonnet-4.5", …) or dspy.LM("openai/gemini-3.5-flash", …). The gateway translates the request and the answer. Keep the openai/ prefix and the same api_base; don't switch to anthropic/ or gemini/.

Streaming

dspy.streamify streams a program's output; it works through the gateway like any Chat Completions stream. A stream from Claude or Gemini arrives as OpenAI chunks.

Tool calls

dspy.ReAct and modules that use dspy.Tool work on every provider the gateway translates to. Each reasoning step and each tool round trip is its own call in Request logs.

def opening_hours(shop: str) -> str:
    return "09:00 to 17:30"

agent = dspy.ReAct("question -> answer", tools=[opening_hours])
print(agent(question="When does the Utrecht shop open?").answer)

Reading what the gateway did

DSPy doesn't show response headers. Look the calls up in Request logs: filter on Key, Model or Path (/v1/chat/completions). lm.history (or dspy.inspect_history()) shows what DSPy sent and received; cost is never in the answer, so read it in Request logs.

Errors

The gateway writes its errors in the OpenAI format, and DSPy raises them as exceptions with the status and message. Common ones while setting up: 401 invalid_api_key, 403 model_not_allowed (your key may not call that model), 404 model_not_found, 400 when the Responses API reaches a non-OpenAI model (set model_type="chat"), and 429 when an optimiser runs into a limit on the key or team. The full list is in Headers and errors.

Model names

The part after openai/ is the name your organisation approved under Models in the Control room. GET https://<your-gateway>/v1/models with your key lists exactly the models it may call.

What goes through the gateway

Feature Through the gateway
All modules, ReAct, optimisers, streaming, with model_type="chat" Yes, to OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models
model_type="responses" (the default for openai/ from DSPy 3.5) OpenAI and Azure OpenAI models only; another provider gets a clear 400
dspy.Embedder with an openai/ embedding model OpenAI and Azure OpenAI embedding models only

Check it works in a minute

python -c "
import dspy
dspy.configure(lm=dspy.LM('openai/gpt-4o-mini', api_base='https://<your-gateway>/v1', api_key='vk_…', model_type='chat', cache=False))
print(dspy.Predict('prompt -> reply')(prompt='Reply with exactly: ok').reply)"

A short answer means the address and key are right. The call shows in Request logs within seconds.

Good to know

  • DSPy caches answers by default: a repeated call never reaches the gateway, so it has no row in Request logs. Pass cache=False while you check the set-up.
  • num_retries defaults to 3. A fallback chain may already have tried another model before you see an error (connection errors, time-outs, 429 and 5xx), so lower it to 1 or 2 to avoid multiplying attempts.
  • Give optimiser runs their own key: compilation calls then show apart from production calls in Request logs and Cost, and a rate limit on that key keeps a large run from starving the rest.
  • DSPy's own docs: Language models.