AnythingLLM¶
Choose Generic OpenAI as AnythingLLM's model provider and point it at the gateway: every chat, document question and agent step is then checked against your policies, charged to a team and recorded in Request logs. Workspaces can each use a different model, from any provider your organisation approved.
What it is¶
AnythingLLM is a chat and document assistant that runs as a desktop app or a self-hosted server with workspaces. Its Generic OpenAI provider talks to any server that speaks the OpenAI format.
The two settings¶
Go to Settings › AI Providers › LLM (the LLM Preference page) and choose Generic OpenAI:
| Field | Value |
|---|---|
| Base URL | https://<your-gateway>/v1 |
| API Key | Your Vidai key (starts vk_), never a provider's key |
| Selected Model | A model your key may call, for example gpt-4o-mini (a list when the gateway answers, otherwise type the name) |
| Model context window | The model's context size, for example 128000 |
| Max Tokens | The longest answer, for example 4096 |
On a self-hosted server you can set the same in the environment instead: LLM_PROVIDER='generic-openai',
GENERIC_OPEN_AI_BASE_PATH, GENERIC_OPEN_AI_API_KEY, GENERIC_OPEN_AI_MODEL_PREF,
GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT and GENERIC_OPEN_AI_MAX_TOKENS.
The gateway's address is under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.
A minimal call¶
- Save the provider settings.
- Open a workspace and start a new thread.
- Send "Reply with exactly: ok". A short answer means the address and key are right.
Streaming¶
AnythingLLM streams answers. A stream from Claude or Gemini arrives as OpenAI chunks; Vidai records the tokens and the cost either way.
Tool calls¶
AnythingLLM's agents (@agent in a thread) use the workspace's model for their reasoning and skills. Each step is its
own call in Request logs.
Reading what the gateway did¶
AnythingLLM doesn't show response headers. Look the call up in Request logs: filter on Key, Model or
Path (/v1/chat/completions). Cost is never in the answer.
Errors¶
AnythingLLM shows the gateway's error message in the thread. Common ones: 401 invalid_api_key, 403
model_not_allowed, 403 content_guardrail_blocked (a guardrail stopped the message), 429 (a limit on the key or
team). The full list is in Headers and errors.
Model names¶
AnythingLLM fills Selected Model from GET /v1/models, which the gateway answers with exactly the models your key
may call, named as your organisation approved them under Models in the Control room. Each workspace can choose
another model in its own settings (Chat Settings › Workspace LLM Provider).
What goes through the gateway¶
| AnythingLLM feature | Through the gateway |
|---|---|
| Chat, document questions, agents, with Generic OpenAI set to the gateway | Yes, to OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models |
| Document embeddings, with the Generic OpenAI embedder set to the gateway | OpenAI and Azure OpenAI embedding models only |
| The built-in embedder, local models, other providers with their own keys | No: they don't pass through the gateway |
Check it works in a minute¶
Send "Reply with exactly: ok" in a workspace, then open Request logs a few seconds later and filter on your key.
Good to know¶
- One key, every user. In a multi-user server every person's chats use the key in the provider settings, so they show under that key and are charged to the team it acts for. AnythingLLM has no per-user key; to split by team, run one instance per team, each with its team's key.
- On the desktop app, each person enters their own key, and their calls show under them.
- If a model rejects
max_tokens, setGENERIC_OPEN_AI_MODEL_MAX_TOKEN_KEY='max_completion_tokens'. - AnythingLLM's own docs: Generic OpenAI.