Haystack¶
A Haystack pipeline or agent calls a model at every generation step. Point the chat generator at the gateway and each of those calls is checked against your policies, charged to the right team and recorded in Request logs, and the same generator can reach Claude and Gemini models as well as OpenAI's.
What it is¶
Haystack (by deepset, package haystack-ai, version 2) builds retrieval, question-answering and agent applications
from components. Two of its chat generators work with the gateway:
OpenAIChatGenerator(in Haystack itself): speaks the OpenAI format at/v1/chat/completions. Through the gateway it reaches OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models.AnthropicChatGenerator(packageanthropic-haystack): speaks the Anthropic format at/v1/messages. Through the gateway it reaches the same models, translated both ways.
The two settings¶
| Generator | Base address | Key | Value |
|---|---|---|---|
OpenAIChatGenerator |
api_base_url, or the OPENAI_BASE_URL environment variable |
api_key (a Secret; by default OPENAI_API_KEY) |
https://<your-gateway>/v1 |
AnthropicChatGenerator |
The ANTHROPIC_BASE_URL environment variable only (the component has no base-address argument) |
api_key (by default ANTHROPIC_API_KEY) |
https://<your-gateway> (no /v1) |
The key is your Vidai key (starts vk_), never a provider's key. Pass it as
Secret.from_env_var("VIDAI_API_KEY") (or leave the default and put the Vidai key in OPENAI_API_KEY) rather than as
a plain string, so it isn't written into a serialised pipeline. The OpenAI generator sends it as
Authorization: Bearer, the Anthropic generator as x-api-key; the gateway accepts both.
AnthropicChatGenerator hands its key to Anthropic's own SDK, which reads ANTHROPIC_BASE_URL when no address is
given; that variable is the only way to point it at the gateway.
The gateway's address, and the base URL for each kind of SDK, are under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.
A minimal call¶
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
llm = OpenAIChatGenerator(
model="gpt-4o-mini",
api_base_url="https://<your-gateway>/v1",
api_key=Secret.from_env_var("VIDAI_API_KEY"),
)
result = llm.run(messages=[ChatMessage.from_user("Say hello in Dutch.")])
print(result["replies"][0].text)
The same code calls Claude or Gemini: put a model name such as claude-sonnet-4.5 in model. The gateway translates
the request for that provider and the answer back into the OpenAI shape, streams and tool calls included. Use the
generator in a pipeline as usual (for example after a ChatPromptBuilder): every generation step is one call.
With the Anthropic generator:
# export ANTHROPIC_BASE_URL="https://<your-gateway>" ANTHROPIC_API_KEY="vk_…"
from haystack_integrations.components.generators.anthropic import AnthropicChatGenerator
llm = AnthropicChatGenerator(model="claude-sonnet-4.5")
Streaming¶
from haystack.components.generators.utils import print_streaming_chunk
llm = OpenAIChatGenerator(
model="gpt-4o-mini",
api_base_url="https://<your-gateway>/v1",
api_key=Secret.from_env_var("VIDAI_API_KEY"),
streaming_callback=print_streaming_chunk,
)
llm.run(messages=[ChatMessage.from_user("Count to five.")])
A stream from another provider arrives in the format the generator speaks. Vidai records the tokens and the cost either way.
Tool calls¶
Give the generator (or a Haystack Agent) tools; the model's tool calls come back in the reply's tool_calls. Every
round trip is a call through the gateway, on any provider it translates to.
from haystack.components.agents import Agent
from haystack.tools import tool
@tool
def get_weather(city: str) -> str:
"""Weather for a city."""
return f"Sunny in {city}"
agent = Agent(chat_generator=llm, tools=[get_weather])
result = agent.run(messages=[ChatMessage.from_user("Weather in Utrecht?")])
print(result["messages"][-1].text)
Reading what the gateway did¶
Haystack doesn't hand you the response headers. Find the calls in Request logs: filter by Key or Model; each generation step and tool round trip is its own call. The model that answered, any routing rule or fallback, and the cost are there.
Errors¶
The gateway writes its errors in the format the generator speaks, so the generator raises the SDK's usual exception
(openai.APIStatusError and its subclasses, or anthropic.APIStatusError). Match on the x-vidai-error-code header,
not the message:
import openai
try:
llm.run(messages=[ChatMessage.from_user("Hi")])
except openai.APIStatusError as e:
print(e.status_code, e.response.headers.get("x-vidai-error-code"))
print(e.response.headers.get("x-vidai-request-id"))
| Status | Common gateway codes |
|---|---|
| 401 | missing_api_key, invalid_api_key, conflicting_auth_headers |
| 403 | api_key_disabled, model_not_allowed, denied_by_routing_rule, content_guardrail_blocked |
| 404 | model_not_found, endpoint_not_found |
| 429 | rate_limit_exceeded, token_limit_exceeded, concurrency_limit_exceeded |
| 5xx | ai_traffic_suspended, fallback_exhausted, upstream_timeout, upstream_unreachable |
The full list is in Headers and errors.
Model names¶
Send the names your organisation has approved, as listed under Models in the Control room (for example
gpt-4o-mini, claude-sonnet-4.5, gemini-3.5-flash). GET /v1/models with your key returns exactly the models it
may call. See Model names.
What goes through the gateway¶
| Feature | Through the gateway |
|---|---|
OpenAIChatGenerator, AnthropicChatGenerator: chat, streaming, tools, agents, images |
Yes, to OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models |
OpenAIResponsesChatGenerator |
OpenAI and Azure OpenAI models only; another provider gets a clear 400 |
OpenAITextEmbedder, OpenAIDocumentEmbedder (api_base_url as above) |
Yes, to OpenAI and Azure OpenAI embedding models only |
AzureOpenAIChatGenerator |
No: it sends the key in an api-key header, which the gateway doesn't accept; use OpenAIChatGenerator |
Check it works in a minute¶
export VIDAI_API_KEY="vk_…"
python -c "from haystack.components.generators.chat import OpenAIChatGenerator as G; from haystack.dataclasses import ChatMessage as M; from haystack.utils import Secret; print(G(model='gpt-4o-mini', api_base_url='https://<your-gateway>/v1', api_key=Secret.from_env_var('VIDAI_API_KEY')).run(messages=[M.from_user('ping')])['replies'][0].text)"
A short answer means the address and key are right. The call shows in Request logs within seconds.
Good to know¶
OpenAIChatGeneratorretries five times by default (max_retries, orOPENAI_MAX_RETRIES). The gateway already tries a fallback model before you see an error; setmax_retries=0or a small number.- The gateway's own 429 has no
Retry-Afterorx-ratelimit-*headers; back off on your side. AnthropicChatGenerator'santhropic_server_tools(bash, text editor) work on Claude models only.