Orchestration
What is best-of-N sampling?
Best-of-N generates several candidate answers and keeps the best. How candidates are chosen, why the selector matters more than N, and when it beats one good prompt.
All explainers · Last reviewed:
Why it works at all
Language model output is sampled, not deterministic. Two runs of the same prompt give different answers, and their quality varies. If the model’s good answers outnumber its bad ones, taking several samples raises the chance that at least one is good.
That is the entire mechanism. It does not make the model smarter; it gives you more chances at the model’s existing ability.
Picking the winner is the hard part
Generating candidates is easy. Choosing between them is the real problem, and it is where most of the technique’s value and most of its failure lives.
Selection by an automated check — does it compile, does it pass the tests, does it satisfy the schema — is by far the most reliable, because the check is independent of the model. Selection by another model is a judge, with all the caveats on that page. Selection by a person is the most accurate and the least scalable.
Choosing N
Returns fall off quickly. Going from one candidate to three is a large improvement; three to ten is a small one at more than three times the cost. Most practical uses sit at three to five.
N multiplies cost exactly. Five candidates is five times the generation tokens, plus whatever the selection costs.
When not to use it
If you have no way to tell a good answer from a bad one, best-of-N cannot help you — you will pick at random from a larger pool and pay more for the privilege. Its natural home is work with an objective check: code, structured output, anything that either parses or does not.
Questions people ask
- Is best-of-N the same as raising temperature?
- No, though they interact. Temperature controls how varied each answer is. Best-of-N is about how many you take and how you choose. Some variety helps, because identical candidates give you nothing to choose between.
- Can I use different models for the candidates?
- Yes, and it often helps — different models fail differently, so the pool is more varied than repeated samples from one. At that point you are close to consensus, with selection instead of agreement.
- Does it help with factual accuracy?
- Only if your selector can detect factual errors. Without an external check you are choosing between confident answers, and confidence is not accuracy.
Try it rather than read about it
Best-of-N is one of ClawAI’s 9 orchestration modes, and every candidate it generates is recorded against the run’s cost.