LlamaIndex¶
Query engines, chat engines and agents in LlamaIndex all call the LLM you set once. Point that LLM at the gateway and every call is checked against your policies, charged to the right team and recorded in Request logs, on OpenAI, Anthropic or Gemini models.
What it is¶
LlamaIndex is a Python framework for retrieval over your own data and for agents. It talks to models through LLM classes, each in its own package. Three work with the gateway:
| Class | Package | Format | Model names it accepts |
|---|---|---|---|
OpenAILike |
llama-index-llms-openai-like |
OpenAI, /v1/chat/completions |
Any: use this one for Claude, Gemini or alias names |
OpenAI |
llama-index-llms-openai |
OpenAI | OpenAI's own model names only |
Anthropic |
llama-index-llms-anthropic |
Anthropic, /v1/messages |
Anthropic's own model names only |
OpenAI and Anthropic look the model name up in their own list (to learn its context window) and stop with
"Unknown model" for any other name. Because the gateway translates, OpenAILike reaches OpenAI, Azure OpenAI,
Anthropic, Gemini and Vertex AI models with whatever names your organisation approved.
The two settings¶
| Class | Base address | Key | Value |
|---|---|---|---|
OpenAILike |
api_base |
api_key |
https://<your-gateway>/v1 |
OpenAI |
api_base, or the OPENAI_API_BASE environment variable |
api_key, or OPENAI_API_KEY |
https://<your-gateway>/v1 |
Anthropic |
base_url |
api_key, or ANTHROPIC_API_KEY |
https://<your-gateway> (no /v1) |
The key is your Vidai key (starts vk_), never a provider's key. Note that LlamaIndex reads OPENAI_API_BASE, not the
OpenAI SDK's OPENAI_BASE_URL. The OpenAI classes send the key as Authorization: Bearer, the Anthropic class as
x-api-key; the gateway accepts both.
The gateway's address, and the base URL for each kind of SDK, are under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.
A minimal call¶
from llama_index.core import Settings
from llama_index.llms.openai_like import OpenAILike
llm = OpenAILike(
model="claude-sonnet-4.5",
api_base="https://<your-gateway>/v1",
api_key="vk_…",
is_chat_model=True, # use /v1/chat/completions
is_function_calling_model=True, # let agents use tools
context_window=200000, # the model's window: LlamaIndex can't look it up
)
Settings.llm = llm # query engines, chat engines and agents use it from here on
print(llm.complete("Say hello in Dutch."))
Set is_chat_model=True: without it OpenAILike sends calls to the legacy /v1/completions, which the gateway serves
for OpenAI models only. Set context_window to the model's real window so LlamaIndex packs retrieved text correctly.
For an OpenAI model name, the OpenAI class needs nothing more:
from llama_index.llms.openai import OpenAI
Settings.llm = OpenAI(model="gpt-4o-mini", api_base="https://<your-gateway>/v1", api_key="vk_…")
Streaming¶
for chunk in llm.stream_complete("Count to five."):
print(chunk.delta, end="")
Query engines stream with index.as_query_engine(streaming=True). A stream from Claude or Gemini arrives as OpenAI
chunks. Vidai records the tokens and the cost either way.
Tool calls¶
Agents call your Python functions as tools; each round trip is a call through the gateway, on any provider it translates to.
import asyncio
from llama_index.core.agent.workflow import FunctionAgent
def get_weather(city: str) -> str:
"""Weather for a city."""
return f"Sunny in {city}"
agent = FunctionAgent(tools=[get_weather], llm=llm)
async def main():
print(await agent.run("Weather in Utrecht?"))
asyncio.run(main())
With OpenAILike, set is_function_calling_model=True, or the agent won't offer tools to the model.
Reading what the gateway did¶
LlamaIndex doesn't hand you the response headers. Find the calls in Request logs: filter by Key or Model; each query, retrieval answer and agent step is its own call. The model that answered, any routing rule or fallback, and the cost are there.
Errors¶
The gateway writes its errors in the format the class speaks, so LlamaIndex raises the SDK's usual exception
(openai.APIStatusError and its subclasses, or anthropic.APIStatusError), after its own retries. Match on the
x-vidai-error-code header, not the message:
import openai
try:
llm.complete("Hi")
except openai.APIStatusError as e:
print(e.status_code, e.response.headers.get("x-vidai-error-code"))
print(e.response.headers.get("x-vidai-request-id"))
| Status | Common gateway codes |
|---|---|
| 401 | missing_api_key, invalid_api_key, conflicting_auth_headers |
| 403 | api_key_disabled, model_not_allowed, denied_by_routing_rule, content_guardrail_blocked |
| 404 | model_not_found, endpoint_not_found |
| 429 | rate_limit_exceeded, token_limit_exceeded, concurrency_limit_exceeded |
| 5xx | ai_traffic_suspended, fallback_exhausted, upstream_timeout, upstream_unreachable |
The full list is in Headers and errors.
Model names¶
Send the names your organisation has approved, as listed under Models in the Control room (for example
gpt-4o-mini, claude-sonnet-4.5, gemini-3.5-flash). GET /v1/models with your key returns exactly the models it
may call. See Model names. The Anthropic class only takes names Anthropic itself uses
(such as claude-sonnet-4-5); for other names use OpenAILike.
What goes through the gateway¶
| Feature | Through the gateway |
|---|---|
Completion, chat, streaming, query and chat engines, agents and tools (OpenAILike, OpenAI, Anthropic) |
Yes, to OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models |
Embeddings with OpenAIEmbedding(api_base=..., api_key=...) |
Yes, to OpenAI and Azure OpenAI embedding models only |
OpenAIResponses |
OpenAI and Azure OpenAI models only; another provider gets a clear 400 |
AzureOpenAI |
No: it sends the key in an api-key header, which the gateway doesn't accept; use OpenAILike |
Embeddings are set apart from the LLM (Settings.embed_model); point that at the gateway too if you want indexing
governed and costed.
Check it works in a minute¶
python -c "from llama_index.llms.openai import OpenAI; print(OpenAI(model='gpt-4o-mini', api_base='https://<your-gateway>/v1', api_key='vk_…', max_tokens=5).complete('ping'))"
A short answer means the address and key are right. The call shows in Request logs within seconds.
Good to know¶
- The OpenAI classes retry three times by default and
Anthropicten times (max_retries). A fallback chain may already have tried another model before you see an error; setmax_retries=0or a small number. - The gateway's own 429 has no
Retry-Afterheader; back off on your side. - Anything other than chat sent to a model of another provider is refused with a 400, never sent somewhere else.