Foundations
What is LLM orchestration?
LLM orchestration is the layer that decides which model runs, in what order, and what happens to the output. How it differs from prompting and from agents.
All explainers · Last reviewed:
It is not prompt engineering
Prompt engineering improves a single call. Orchestration decides how many calls there are, which models make them, and how their outputs combine. You can have excellent prompts and no orchestration, and the result is a system that fails the moment one provider has a bad hour.
The distinction matters because the two are optimised differently. A better prompt is cheap and improves quality slightly. Better orchestration costs tokens and improves reliability substantially.
What an orchestration layer decides
Which model. Whether to ask more than one. Whether to check the answer before returning it. What to do on a refusal, a timeout, or a rate limit. Whether the output of this step becomes the input of the next. Whether the whole thing is affordable before it starts.
Each of those is a policy, and each can be wrong independently. That is why orchestration is worth naming as its own layer rather than scattering the decisions through application code.
The common techniques
Routing sends a request to an appropriate model. Fallback handles failure. Consensus asks several models and looks at agreement. Best-of-N generates candidates and keeps one. A judge scores answers. Verification checks a claim against something outside the model. Pipelines chain steps. Task decomposition splits a large request into smaller ones.
ClawAI implements nine of these as separate orchestration modes, plus judge and compare as their own surfaces. Each has a page here explaining what it is before you decide whether you want it.
When not to orchestrate
Orchestration multiplies cost and latency. A three-model consensus is roughly three times the tokens and as slow as the slowest model. For a question whose answer you can check at a glance, that is a bad trade.
The heuristic that holds up: orchestrate when being wrong is expensive and checking is hard. Otherwise send one request to one model and read the answer.
Questions people ask
- Is orchestration the same as an agent framework?
- Overlapping but not identical. An agent decides its own next step, often with tools. Orchestration is the surrounding policy — which model, how many, what on failure — and applies just as much to a workflow with no agent in it.
- Does orchestration need a framework?
- No. A retry with a different model is orchestration. Frameworks help when the policies get numerous enough that you would otherwise be reimplementing them per feature.
- How much does it cost?
- In tokens, roughly proportional to how many model calls the policy makes. A single routed call costs about the same as an unrouted one; consensus over three models costs about three times as much. The cost is predictable, which is what makes it a budgeting decision rather than a gamble.
Try it rather than read about it
ClawAI runs 9 orchestration modes alongside ordinary chat, and records which models a run used, so the cost of a technique is visible rather than inferred.