Skip to content

Migrate from calling providers directly

Purpose: move an existing app onto the gateway with two changed settings, then remove what it no longer needs: provider keys in your secrets, your own failover between providers, your own cost logging. Your code, prompts and SDK stay as they are.

Before you start

Change two settings

 client = OpenAI(
-    api_key=os.environ["OPENAI_API_KEY"],
+    base_url=os.environ["VIDAI_GATEWAY"] + "/v1",
+    api_key=os.environ["VIDAI_KEY"],
 )
 client = Anthropic(
-    api_key=os.environ["ANTHROPIC_API_KEY"],
+    base_url=os.environ["VIDAI_GATEWAY"],
+    api_key=os.environ["VIDAI_KEY"],
 )
 client = genai.Client(
-    api_key=os.environ["GEMINI_API_KEY"],
+    api_key=os.environ["VIDAI_KEY"],
+    http_options=types.HttpOptions(base_url=os.environ["VIDAI_GATEWAY"]),
 )

Many SDKs and tools read their address and key from the environment, so no code changes at all:

# OpenAI SDKs
export OPENAI_BASE_URL="https://<your-gateway>/v1"
export OPENAI_API_KEY="$VIDAI_KEY"

# Anthropic SDKs
export ANTHROPIC_BASE_URL="https://<your-gateway>"
export ANTHROPIC_API_KEY="$VIDAI_KEY"

What you can delete

You had Now Remove
A provider key per app, in each app's secrets One Vidai key per app; the providers' keys stay with the gateway The provider keys from the app's secrets, then rotate them at the provider
A client per provider, to reach models from different vendors One SDK reaches every model your key may call (Any SDK to any provider) The extra clients, if you like
Your own failover from one provider to another Fallback chains your organisation sets up, applied before you see a failure Your cross-provider failover code
Your own token and cost logging Every call priced in Request logs and Money › Cost The cost bookkeeping
Your own filters for personal data in prompts Guardrails, applied to every call Duplicated filters, once your organisation's Guardrails cover them
A model name hard-coded per provider A name your organisation can move (Model names and aliases) Nothing: keep the name, drop the per-provider switch

Retries

The gateway can try the next model in a fallback chain when a provider fails: on a connection error, a time-out, 429 or 5xx, before anything has reached you, at most 3 attempts in all. If the chain runs out, you get the last model's own error. x-vidai-fallback-attempts on the answer says how many attempts were made. It does this only where an admin set a chain to Try the next model (Fallback and retries).

So keep your SDK's own retries few and spaced out (the SDKs default to two, with backoff), and never retry a 401 or 403. Two layers of eager retries multiply: three of yours times three of the gateway's is nine calls to a struggling provider, and a provider may charge for an attempt that failed after it received the call.

Check it worked

  1. Run the app's usual path once.
  2. Open Daily › Request logs and filter by the app's Key: every call is there, with its model, outcome and cost.
  3. Check x-vidai-adapted on a few answers if a model now sits behind another provider: it names anything that couldn't be carried exactly.

Request logs filtered to one key's calls, each with its model, outcome, time and cost

Good to know

  • x-vidai-* headers are extra: a client that ignores them works the same.
  • The gateway doesn't send rate-limit headers of its own; a provider's pass through and describe the provider's limits, not your key's (Limits).
  • Files, batches and uploads live in the provider's account the gateway uses, shared by every key on that provider (Endpoints).
  • The Responses API, embeddings, images and audio work with a provider of the same format only.