Skip to content

Databricks

Data teams in Databricks call models from SQL, notebooks and agents, often over your most sensitive tables. Serve the gateway as an external model endpoint and those calls are checked against your policies, charged to the right team and recorded in Request logs, with Claude and Gemini behind the same endpoint as OpenAI's models.

What it is

Databricks Model Serving can serve external models: an endpoint in your workspace that forwards to a model hosted elsewhere. Created with the OpenAI provider and the gateway's address, it sends every call to the gateway. Anything in Databricks that queries a serving endpoint then goes through it: ai_query() in SQL, the AI Playground, notebooks and agents you build.

Genie (AI/BI Genie) is not one of them: Databricks runs Genie on models it manages, and a Genie space can't be pointed at a serving endpoint. The same holds for Databricks Assistant.

The two settings

Create a serving endpoint (in Serving, Create serving endpoint, or with the API) with an external model:

Setting Value
Provider OpenAI
Task Chat (llm/v1/chat)
Model name A name your organisation has approved, for example gpt-4o-mini
openai_api_base https://<your-gateway>/v1
openai_api_key A Databricks secret holding your Vidai key (starts vk_), never a provider's key
openai_api_type, openai_api_version, openai_organization Leave empty

Store the key in a secret scope first (databricks secrets put-secret vidai key) and refer to it as {{secrets/vidai/key}}. The same endpoint configuration as JSON:

{
  "name": "vidai",
  "config": {
    "served_entities": [{
      "name": "gpt-4o-mini",
      "external_model": {
        "name": "gpt-4o-mini",
        "provider": "openai",
        "task": "llm/v1/chat",
        "openai_config": {
          "openai_api_base": "https://<your-gateway>/v1",
          "openai_api_key": "{{secrets/vidai/key}}"
        }
      }
    }]
  }
}

The gateway's address is under Connect your app on the Keys page and in System › Settings › Gateway. You get the key from Keys › Issue a key; its secret is shown once.

A minimal call

SELECT ai_query('vidai', 'Reply with exactly: ok');
from openai import OpenAI

client = OpenAI(
    base_url="https://<your-workspace>/serving-endpoints",
    api_key="<a Databricks token>",  # Databricks' own token: Databricks adds your Vidai key
)
reply = client.chat.completions.create(
    model="vidai",  # the serving endpoint's name
    messages=[{"role": "user", "content": "Reply with exactly: ok"}],
)
print(reply.choices[0].message.content)

To serve Claude or Gemini, add a served entity (or create another endpoint) with the external model name claude-sonnet-4.5 or gemini-3.5-flash, still with the OpenAI provider and the gateway's address. The gateway translates the request for that provider and the answer back into the OpenAI shape Databricks expects.

Streaming

Streaming through the endpoint (stream=True with the OpenAI SDK) comes from the gateway in OpenAI chunks, whichever provider answered. ai_query() returns whole answers.

Tool calls

Agents you build in Databricks pass tools through the serving endpoint as in the OpenAI format; they work on every provider the gateway translates to. In Request logs, the Tool filter under More filters finds the calls that offered or called a tool by its name.

Reading what the gateway did

Databricks doesn't pass the gateway's response headers on. Find the calls in Request logs: filter by Key (the endpoint's key) or Model under More filters. Each call shows the model that answered, its cost, and which policy or rule acted on it.

Errors

Databricks returns the gateway's refusals as errors from the endpoint, with the gateway's message. The common ones:

Status Gateway code What to do
401 invalid_api_key, missing_api_key Check the secret the endpoint uses
403 model_not_allowed, denied_by_routing_rule Serve a model the key may call
403 content_guardrail_blocked A guardrail stopped the content, for example personal data from a table
404 model_not_found Check the external model name against Models
429 rate_limit_exceeded, token_limit_exceeded, concurrency_limit_exceeded An ai_query() over many rows went too fast: raise the key's limits or query fewer rows at a time

Model names

The external model's name is what Databricks sends to the gateway: use the names your organisation has approved, as listed under Models in the Control room (aliases work too). Callers in Databricks use the serving endpoint's name.

What goes through the gateway

In Databricks Through the gateway
ai_query(), the AI Playground, notebooks and agents that query the external model endpoint Yes, to every provider your organisation has set up
Embeddings served the same way (task llm/v1/embeddings) To OpenAI and Azure OpenAI models only
Genie, Databricks Assistant and Databricks-hosted foundation models No: Databricks calls its own models

Check it works in a minute

Run SELECT ai_query('vidai', 'Reply with exactly: ok'); in a SQL editor. ok means the endpoint reaches the gateway. The call shows in Request logs within seconds.

Good to know

  • Databricks calls the gateway from its own network (the serving infrastructure): the gateway must be reachable from it.
  • Every caller of an endpoint shares its key, so spend is split per endpoint, not per person. Give each workspace or team its own endpoint and key (an agent's key in Vidai works well) to keep their spend apart.
  • Rate limits and usage tracking that Databricks can add to an endpoint count only what passes through it; Vidai's limits and Request logs cover every caller, inside and outside Databricks.
  • ai_query() over a large table sends one call per row: check the key's rate and token limits before you run it.