Skip to content

Trial two models with an A/B split

Purpose: "is the cheaper model good enough?" is answered on your own traffic, not a benchmark. An A/B experiment splits one model's calls between two arms by the share you set. Your apps don't change, and the experiment's page compares the arms: calls, failures, typical and slowest latency, tokens, and cost in each currency.

In the demo world, the RFP assistant application asks for claude-sonnet-4-6. You send 20% of its calls to gpt-4.1 to see whether its pitch answers are as good, for less; the steps below set that up.

Before you start

  • You're an organisation admin, or a group admin for your own teams.
  • Both models are ones the keys at that level may call.

Steps

  1. Open Models & routing › Routing and choose New routing… › Split traffic between two (A/B) › Continue (or New A/B experiment on the A/B experiments tab).
  2. Applies to: the level and which one, for example Application › RFP assistant.
  3. Calls to: the provider and the model calls ask for.
  4. First arm and Second arm: for each, the provider first, then the model. Either arm may be the model as asked.
  5. Set First arm's share of calls with the slider: it moves in whole percents and reads, for example, "First 80% · second 20%". Start small with the newcomer.
  6. Add Why (and an end date if you like).
  7. If some keys can't use an arm, choose Skip these N keys (N is how many): they keep the model they ask for.
  8. Check what it changes, then Save.

Setting up an A/B split on an application: the model asked for, two arms and the share slider

Check it worked

The experiment is on the A/B experiments tab, running. Its page shows results per arm as calls arrive. In Request logs its calls say Routed, and Why this happened names the experiment.

An A/B experiment's results per arm: calls, failures, typical and slowest latency, tokens and cost

Change the split, or end it

  • Change the share: open the experiment, choose Change, move the slider, Check what it changes and Save changes.
  • Pick a winner: delete the experiment and make a routing rule to the winning arm (or none, if the model asked for won).
  • Pause it for a while with Pause… and a reason.

Good to know

  • The share is a whole number from 1 to 99; the second arm gets the rest.
  • An experiment that loses an arm (a model switched off, say) is shut, and doesn't come back on by itself.
  • Cost per arm is shown per currency, never converted.
  • Answer quality isn't compared: the figures are calls, failures, speed, tokens and cost.