Trial two models with an A/B split¶
Purpose: "is the cheaper model good enough?" is answered on your own traffic, not a benchmark. An A/B experiment splits one model's calls between two arms by the share you set. Your apps don't change, and the experiment's page compares the arms: calls, failures, typical and slowest latency, tokens, and cost in each currency.
In the demo world, the RFP assistant application asks for claude-sonnet-4-6. You send 20% of its calls to gpt-4.1
to see whether its pitch answers are as good, for less; the steps below set that up.
Before you start¶
- You're an organisation admin, or a group admin for your own teams.
- Both models are ones the keys at that level may call.
Steps¶
- Open Models & routing › Routing and choose New routing… › Split traffic between two (A/B) › Continue (or New A/B experiment on the A/B experiments tab).
- Applies to: the level and which one, for example Application › RFP assistant.
- Calls to: the provider and the model calls ask for.
- First arm and Second arm: for each, the provider first, then the model. Either arm may be the model as asked.
- Set First arm's share of calls with the slider: it moves in whole percents and reads, for example, "First 80% · second 20%". Start small with the newcomer.
- Add Why (and an end date if you like).
- If some keys can't use an arm, choose Skip these N keys (N is how many): they keep the model they ask for.
- Check what it changes, then Save.
Check it worked¶
The experiment is on the A/B experiments tab, running. Its page shows results per arm as calls arrive. In Request logs its calls say Routed, and Why this happened names the experiment.

Change the split, or end it¶
- Change the share: open the experiment, choose Change, move the slider, Check what it changes and Save changes.
- Pick a winner: delete the experiment and make a routing rule to the winning arm (or none, if the model asked for won).
- Pause it for a while with Pause… and a reason.
Related¶
- Send a model's calls to another model
- See which routing rule wins: an experiment beats a default rule, not a required one.
Good to know¶
- The share is a whole number from 1 to 99; the second arm gets the rest.
- An experiment that loses an arm (a model switched off, say) is shut, and doesn't come back on by itself.
- Cost per arm is shown per currency, never converted.
- Answer quality isn't compared: the figures are calls, failures, speed, tokens and cost.