Skip to content

Haystack

A Haystack pipeline or agent calls a model at every generation step. Point the chat generator at the gateway and each of those calls is checked against your policies, charged to the right team and recorded in Request logs, and the same generator can reach Claude and Gemini models as well as OpenAI's.

What it is

Haystack (by deepset, package haystack-ai, version 2) builds retrieval, question-answering and agent applications from components. Two of its chat generators work with the gateway:

  • OpenAIChatGenerator (in Haystack itself): speaks the OpenAI format at /v1/chat/completions. Through the gateway it reaches OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models.
  • AnthropicChatGenerator (package anthropic-haystack): speaks the Anthropic format at /v1/messages. Through the gateway it reaches the same models, translated both ways.

The two settings

Generator Base address Key Value
OpenAIChatGenerator api_base_url, or the OPENAI_BASE_URL environment variable api_key (a Secret; by default OPENAI_API_KEY) https://<your-gateway>/v1
AnthropicChatGenerator The ANTHROPIC_BASE_URL environment variable only (the component has no base-address argument) api_key (by default ANTHROPIC_API_KEY) https://<your-gateway> (no /v1)

The key is your Vidai key (starts vk_), never a provider's key. Pass it as Secret.from_env_var("VIDAI_API_KEY") (or leave the default and put the Vidai key in OPENAI_API_KEY) rather than as a plain string, so it isn't written into a serialised pipeline. The OpenAI generator sends it as Authorization: Bearer, the Anthropic generator as x-api-key; the gateway accepts both.

AnthropicChatGenerator hands its key to Anthropic's own SDK, which reads ANTHROPIC_BASE_URL when no address is given; that variable is the only way to point it at the gateway.

The gateway's address, and the base URL for each kind of SDK, are under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.

A minimal call

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret

llm = OpenAIChatGenerator(
    model="gpt-4o-mini",
    api_base_url="https://<your-gateway>/v1",
    api_key=Secret.from_env_var("VIDAI_API_KEY"),
)
result = llm.run(messages=[ChatMessage.from_user("Say hello in Dutch.")])
print(result["replies"][0].text)

The same code calls Claude or Gemini: put a model name such as claude-sonnet-4.5 in model. The gateway translates the request for that provider and the answer back into the OpenAI shape, streams and tool calls included. Use the generator in a pipeline as usual (for example after a ChatPromptBuilder): every generation step is one call.

With the Anthropic generator:

# export ANTHROPIC_BASE_URL="https://<your-gateway>"  ANTHROPIC_API_KEY="vk_…"
from haystack_integrations.components.generators.anthropic import AnthropicChatGenerator

llm = AnthropicChatGenerator(model="claude-sonnet-4.5")

Streaming

from haystack.components.generators.utils import print_streaming_chunk

llm = OpenAIChatGenerator(
    model="gpt-4o-mini",
    api_base_url="https://<your-gateway>/v1",
    api_key=Secret.from_env_var("VIDAI_API_KEY"),
    streaming_callback=print_streaming_chunk,
)
llm.run(messages=[ChatMessage.from_user("Count to five.")])

A stream from another provider arrives in the format the generator speaks. Vidai records the tokens and the cost either way.

Tool calls

Give the generator (or a Haystack Agent) tools; the model's tool calls come back in the reply's tool_calls. Every round trip is a call through the gateway, on any provider it translates to.

from haystack.components.agents import Agent
from haystack.tools import tool

@tool
def get_weather(city: str) -> str:
    """Weather for a city."""
    return f"Sunny in {city}"

agent = Agent(chat_generator=llm, tools=[get_weather])
result = agent.run(messages=[ChatMessage.from_user("Weather in Utrecht?")])
print(result["messages"][-1].text)

Reading what the gateway did

Haystack doesn't hand you the response headers. Find the calls in Request logs: filter by Key or Model; each generation step and tool round trip is its own call. The model that answered, any routing rule or fallback, and the cost are there.

Errors

The gateway writes its errors in the format the generator speaks, so the generator raises the SDK's usual exception (openai.APIStatusError and its subclasses, or anthropic.APIStatusError). Match on the x-vidai-error-code header, not the message:

import openai

try:
    llm.run(messages=[ChatMessage.from_user("Hi")])
except openai.APIStatusError as e:
    print(e.status_code, e.response.headers.get("x-vidai-error-code"))
    print(e.response.headers.get("x-vidai-request-id"))
Status Common gateway codes
401 missing_api_key, invalid_api_key, conflicting_auth_headers
403 api_key_disabled, model_not_allowed, denied_by_routing_rule, content_guardrail_blocked
404 model_not_found, endpoint_not_found
429 rate_limit_exceeded, token_limit_exceeded, concurrency_limit_exceeded
5xx ai_traffic_suspended, fallback_exhausted, upstream_timeout, upstream_unreachable

The full list is in Headers and errors.

Model names

Send the names your organisation has approved, as listed under Models in the Control room (for example gpt-4o-mini, claude-sonnet-4.5, gemini-3.5-flash). GET /v1/models with your key returns exactly the models it may call. See Model names.

What goes through the gateway

Feature Through the gateway
OpenAIChatGenerator, AnthropicChatGenerator: chat, streaming, tools, agents, images Yes, to OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models
OpenAIResponsesChatGenerator OpenAI and Azure OpenAI models only; another provider gets a clear 400
OpenAITextEmbedder, OpenAIDocumentEmbedder (api_base_url as above) Yes, to OpenAI and Azure OpenAI embedding models only
AzureOpenAIChatGenerator No: it sends the key in an api-key header, which the gateway doesn't accept; use OpenAIChatGenerator

Check it works in a minute

export VIDAI_API_KEY="vk_…"
python -c "from haystack.components.generators.chat import OpenAIChatGenerator as G; from haystack.dataclasses import ChatMessage as M; from haystack.utils import Secret; print(G(model='gpt-4o-mini', api_base_url='https://<your-gateway>/v1', api_key=Secret.from_env_var('VIDAI_API_KEY')).run(messages=[M.from_user('ping')])['replies'][0].text)"

A short answer means the address and key are right. The call shows in Request logs within seconds.

Good to know

  • OpenAIChatGenerator retries five times by default (max_retries, or OPENAI_MAX_RETRIES). The gateway already tries a fallback model before you see an error; set max_retries=0 or a small number.
  • The gateway's own 429 has no Retry-After or x-ratelimit-* headers; back off on your side.
  • AnthropicChatGenerator's anthropic_server_tools (bash, text editor) work on Claude models only.