Skip to content

AnythingLLM

Choose Generic OpenAI as AnythingLLM's model provider and point it at the gateway: every chat, document question and agent step is then checked against your policies, charged to a team and recorded in Request logs. Workspaces can each use a different model, from any provider your organisation approved.

What it is

AnythingLLM is a chat and document assistant that runs as a desktop app or a self-hosted server with workspaces. Its Generic OpenAI provider talks to any server that speaks the OpenAI format.

The two settings

Go to Settings › AI Providers › LLM (the LLM Preference page) and choose Generic OpenAI:

Field Value
Base URL https://<your-gateway>/v1
API Key Your Vidai key (starts vk_), never a provider's key
Selected Model A model your key may call, for example gpt-4o-mini (a list when the gateway answers, otherwise type the name)
Model context window The model's context size, for example 128000
Max Tokens The longest answer, for example 4096

On a self-hosted server you can set the same in the environment instead: LLM_PROVIDER='generic-openai', GENERIC_OPEN_AI_BASE_PATH, GENERIC_OPEN_AI_API_KEY, GENERIC_OPEN_AI_MODEL_PREF, GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT and GENERIC_OPEN_AI_MAX_TOKENS.

The gateway's address is under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.

A minimal call

  1. Save the provider settings.
  2. Open a workspace and start a new thread.
  3. Send "Reply with exactly: ok". A short answer means the address and key are right.

Streaming

AnythingLLM streams answers. A stream from Claude or Gemini arrives as OpenAI chunks; Vidai records the tokens and the cost either way.

Tool calls

AnythingLLM's agents (@agent in a thread) use the workspace's model for their reasoning and skills. Each step is its own call in Request logs.

Reading what the gateway did

AnythingLLM doesn't show response headers. Look the call up in Request logs: filter on Key, Model or Path (/v1/chat/completions). Cost is never in the answer.

Errors

AnythingLLM shows the gateway's error message in the thread. Common ones: 401 invalid_api_key, 403 model_not_allowed, 403 content_guardrail_blocked (a guardrail stopped the message), 429 (a limit on the key or team). The full list is in Headers and errors.

Model names

AnythingLLM fills Selected Model from GET /v1/models, which the gateway answers with exactly the models your key may call, named as your organisation approved them under Models in the Control room. Each workspace can choose another model in its own settings (Chat Settings › Workspace LLM Provider).

What goes through the gateway

AnythingLLM feature Through the gateway
Chat, document questions, agents, with Generic OpenAI set to the gateway Yes, to OpenAI, Azure OpenAI, Anthropic, Gemini and Vertex AI models
Document embeddings, with the Generic OpenAI embedder set to the gateway OpenAI and Azure OpenAI embedding models only
The built-in embedder, local models, other providers with their own keys No: they don't pass through the gateway

Check it works in a minute

Send "Reply with exactly: ok" in a workspace, then open Request logs a few seconds later and filter on your key.

Good to know

  • One key, every user. In a multi-user server every person's chats use the key in the provider settings, so they show under that key and are charged to the team it acts for. AnythingLLM has no per-user key; to split by team, run one instance per team, each with its team's key.
  • On the desktop app, each person enters their own key, and their calls show under them.
  • If a model rejects max_tokens, set GENERIC_OPEN_AI_MODEL_MAX_TOKEN_KEY='max_completion_tokens'.
  • AnythingLLM's own docs: Generic OpenAI.