Skip to content

LlamaIndex

Query engines, chat engines and agents in LlamaIndex all call the LLM you set once. Point that LLM at the gateway and every call is checked against your policies, charged to the right team and recorded in Request logs, on OpenAI, Anthropic or Gemini models.

What it is

LlamaIndex is a Python framework for retrieval over your own data and for agents. It talks to models through LLM classes, each in its own package. Three work with the gateway:

Class Package Format Model names it accepts
OpenAILike llama-index-llms-openai-like OpenAI, /v1/chat/completions Any: use this one for Claude, Gemini or alias names
OpenAI llama-index-llms-openai OpenAI OpenAI's own model names only
Anthropic llama-index-llms-anthropic Anthropic, /v1/messages Anthropic's own model names only

OpenAI and Anthropic look the model name up in their own list (to learn its context window) and stop with "Unknown model" for any other name. Because the gateway translates, OpenAILike reaches OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models with whatever names your organisation approved.

The two settings

Class Base address Key Value
OpenAILike api_base api_key https://<your-gateway>/v1
OpenAI api_base, or the OPENAI_API_BASE environment variable api_key, or OPENAI_API_KEY https://<your-gateway>/v1
Anthropic base_url api_key, or ANTHROPIC_API_KEY https://<your-gateway> (no /v1)

The key is your Vidai key (starts vk_), never a provider's key. Note that LlamaIndex reads OPENAI_API_BASE, not the OpenAI SDK's OPENAI_BASE_URL. The OpenAI classes send the key as Authorization: Bearer, the Anthropic class as x-api-key; the gateway accepts both.

The gateway's address, and the base URL for each kind of SDK, are under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.

A minimal call

from llama_index.core import Settings
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="claude-sonnet-4.5",
    api_base="https://<your-gateway>/v1",
    api_key="vk_…",
    is_chat_model=True,              # use /v1/chat/completions
    is_function_calling_model=True,  # let agents use tools
    context_window=200000,           # the model's window: LlamaIndex can't look it up
)
Settings.llm = llm  # query engines, chat engines and agents use it from here on

print(llm.complete("Say hello in Dutch."))

Set is_chat_model=True: without it OpenAILike sends calls to the legacy /v1/completions, which the gateway serves for OpenAI models only. Set context_window to the model's real window so LlamaIndex packs retrieved text correctly.

For an OpenAI model name, the OpenAI class needs nothing more:

from llama_index.llms.openai import OpenAI

Settings.llm = OpenAI(model="gpt-4o-mini", api_base="https://<your-gateway>/v1", api_key="vk_…")

Streaming

for chunk in llm.stream_complete("Count to five."):
    print(chunk.delta, end="")

Query engines stream with index.as_query_engine(streaming=True). A stream from Claude or Gemini arrives as OpenAI chunks. Vidai records the tokens and the cost either way.

Tool calls

Agents call your Python functions as tools; each round trip is a call through the gateway, on any provider it translates to.

import asyncio

from llama_index.core.agent.workflow import FunctionAgent

def get_weather(city: str) -> str:
    """Weather for a city."""
    return f"Sunny in {city}"

agent = FunctionAgent(tools=[get_weather], llm=llm)

async def main():
    print(await agent.run("Weather in Utrecht?"))

asyncio.run(main())

With OpenAILike, set is_function_calling_model=True, or the agent won't offer tools to the model.

Reading what the gateway did

LlamaIndex doesn't hand you the response headers. Find the calls in Request logs: filter by Key or Model; each query, retrieval answer and agent step is its own call. The model that answered, any routing rule or fallback, and the cost are there.

Errors

The gateway writes its errors in the format the class speaks, so LlamaIndex raises the SDK's usual exception (openai.APIStatusError and its subclasses, or anthropic.APIStatusError), after its own retries. Match on the x-vidai-error-code header, not the message:

import openai

try:
    llm.complete("Hi")
except openai.APIStatusError as e:
    print(e.status_code, e.response.headers.get("x-vidai-error-code"))
    print(e.response.headers.get("x-vidai-request-id"))
Status Common gateway codes
401 missing_api_key, invalid_api_key, conflicting_auth_headers
403 api_key_disabled, model_not_allowed, denied_by_routing_rule, content_guardrail_blocked
404 model_not_found, endpoint_not_found
429 rate_limit_exceeded, token_limit_exceeded, concurrency_limit_exceeded
5xx ai_traffic_suspended, fallback_exhausted, upstream_timeout, upstream_unreachable

The full list is in Headers and errors.

Model names

Send the names your organisation has approved, as listed under Models in the Control room (for example gpt-4o-mini, claude-sonnet-4.5, gemini-3.5-flash). GET /v1/models with your key returns exactly the models it may call. See Model names. The Anthropic class only takes names Anthropic itself uses (such as claude-sonnet-4-5); for other names use OpenAILike.

What goes through the gateway

Feature Through the gateway
Completion, chat, streaming, query and chat engines, agents and tools (OpenAILike, OpenAI, Anthropic) Yes, to OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models
Embeddings with OpenAIEmbedding(api_base=..., api_key=...) Yes, to OpenAI and Azure OpenAI embedding models only
OpenAIResponses OpenAI and Azure OpenAI models only; another provider gets a clear 400
AzureOpenAI No: it sends the key in an api-key header, which the gateway doesn't accept; use OpenAILike

Embeddings are set apart from the LLM (Settings.embed_model); point that at the gateway too if you want indexing governed and costed.

Check it works in a minute

python -c "from llama_index.llms.openai import OpenAI; print(OpenAI(model='gpt-4o-mini', api_base='https://<your-gateway>/v1', api_key='vk_…', max_tokens=5).complete('ping'))"

A short answer means the address and key are right. The call shows in Request logs within seconds.

Good to know

  • The OpenAI classes retry three times by default and Anthropic ten times (max_retries). A fallback chain may already have tried another model before you see an error; set max_retries=0 or a small number.
  • The gateway's own 429 has no Retry-After header; back off on your side.
  • Anything other than chat sent to a model of another provider is refused with a 400, never sent somewhere else.