Multi-Agent Patterns: Router, Planner and Executor, Critic, Parallel Fan-Out

By the end of this lesson you will be able to look at a multi-agent design and say, before building it, roughly how many tokens it will spend per task compared with one agent doing the whole thing; and you will be able to name, for each of the five common shapes, the one condition that makes it worth its cost.

Part 24 gave you the vocabulary: an agent is a loop, a workflow is a fixed path, and Anthropic’s essay names five workflow patterns, of which this lesson takes the ones that recur in local systems. What that essay does not do, because it was not written for people paying in their own electricity, is the arithmetic. On a hosted model, a second agent is a line item. On your machine it is a second prefill through a model that is already the bottleneck, and possibly a second model resident in memory that the first one now has to share with.

The arithmetic everything else is measured against

Write the single-loop cost symbolically first. These symbols recur through the whole lesson.

Pseudocode — not a real command

S   = system prompt + tool schemas, in tokens          (resent every turn)
Q   = the task, in tokens                              (resent every turn)
O   = tokens the model writes per turn
R   = tokens a tool result adds per turn
n   = number of turns before the loop stops

Input tokens on turn t   = S + Q + (t - 1) * (O + R)

Total input over n turns = n * (S + Q) + (O + R) * n * (n - 1) / 2
Total output over n turns = n * O

The term to stare at is n * (n - 1) / 2. A loop’s input cost grows with the square of the number of turns, because every turn resends everything that came before it. Doubling the number of turns roughly quadruples the reading the model has to do. This is the same fact Part 24’s context-engineering lesson approaches from the memory side; here it is the economics, and it is what every pattern below is either fighting or paying.

Two consequences follow immediately, and both are counter-intuitive:

• A pattern that adds model calls but shortens each context can be cheaper than one long loop. Splitting n turns across m independent contexts turns one n² term into m terms of (n/m)², which totals n²/m.
• A pattern that adds a model call without shortening anything is pure addition. A critic that reads the whole transcript costs what the whole transcript costs, every time it runs.

!Important

Tokens are not the only currency, and on local hardware they are not even the dominant one. A token read by a 1.7B router and a token read by a 235B mixture-of-experts model cost wildly different amounts of time and electricity. So every total below should be weighted: give each model a factor w from your own cost per million tokens in Part 23, and compare w * tokens, not tokens. A pattern that moves work from a large model to a small one can raise the token count and lower the bill at the same time.

Router: a small model spending a little to save a lot

A router is one model call that classifies the task and picks which specialist runs. It is Anthropic’s routing pattern, whose stated condition is that “there are distinct categories that are better handled separately”.

Router

1. 1Small model reads the task and the route listA short prompt: the task, three or four route names, one sentence each. No tools, no documents, no transcript.
2. 2It returns one route and a reasonA structured output with a fixed set of allowed values, so an answer outside the set is caught by the validator rather than by a specialist that was never meant to run.
3. 3The program runs exactly one specialistThe saving is the specialists that did not run. This is the whole point of the pattern.
4. 4On an invalid or failed route, run the safe defaultUsually: run every specialist, or run the one that can answer most things. A router failure must degrade to expensive, never to wrong.

The router is the only place in a well-built system where a small model's opinion decides what a large model does.

Pseudocode — not a real command

Cost without a router  = k * (one specialist run)      k = number of specialists
Cost with a router     = (router call) + 1 * (one specialist run)

Router call  ≈ S_r + Q + O_r,  where S_r is a few hundred tokens and O_r a few dozen
Break-even   : the router pays for itself whenever k > 1 and it is right more often
               than the cost of being wrong

Two design rules fall out of that break-even line.

Make the router’s prompt tiny and its output typed. It sees no documents, no tool schemas and no transcript. S_r should be a few hundred tokens, which is why a 1.7B or 4B model can do it and why it can run on a machine that is not the one serving the big model.

Make being wrong cheap. A router that routes a document question to the tool agent should end with the tool agent saying it found nothing, not with a confident wrong answer. The reference implementation in this part’s project falls back to running both specialists when the router fails or returns an unknown route: expensive, correct, and visible in the trajectory.

✕Common mistake

Routing with the same model that answers. If the router is the 30B model, the router call costs a large-model prefill and you have bought nothing but latency. The pattern is only economical when the router is genuinely small. Serve it as its own alias so the gateway can keep it resident while the large model swaps.

Planner and executor: breaking the quadratic

A planner reads the task once and emits a list of steps. Each step is then executed in its own short context, which never sees the other steps’ transcripts. This is Anthropic’s orchestrator-workers pattern, recommended for cases where “you can’t predict the subtasks needed”.

Planner and executor

1. 1Planner sees the task and the tool listOne call. It writes m steps, each a self-contained instruction with everything the executor needs to do it.
2. 2Each step runs in a fresh contextThe executor gets the step, the tools, and nothing else. This is the step that breaks the quadratic term.
3. 3Results are collected outside the modelYour program holds the list of step results. They do not accumulate in any transcript unless you put them there.
4. 4A final call turns the results into an answerInput is m short results rather than one long transcript.
5. 5Replan only on failureA plan revised after every step is not a plan, it is a loop with extra calls.

The saving is not the planner. The saving is that m short contexts cost less than one long one.

Pseudocode — not a real command

One loop of n turns      : input ≈ n * (S + Q) + (O + R) * n * (n - 1) / 2

Planner + m executors    : input ≈ (S_p + Q) + m * [ (n/m) * (S + Q_step)
                                    + (O + R) * (n/m) * (n/m - 1) / 2 ]
                                 + (S + m * result_size)

The quadratic term falls from  (O + R) * n² / 2  to  (O + R) * n² / (2m)
The linear term rises, because S is resent in every executor context

So the pattern trades a larger linear term for a smaller quadratic one, and it wins exactly when n is large enough for the quadratic term to dominate. For a three-turn task it loses. For a twenty-turn task on a small context budget it is often the difference between finishing and running out of context.

The failure mode is a plan written before anything is known. A planner that has not yet searched anything cannot write step four sensibly, so the useful version replans when a step fails and otherwise leaves the plan alone.

Critic and verifier loops: paying a multiple for a chance at better

A critic reads the answer and says what is wrong with it; the generator tries again. This is the evaluator-optimiser pattern, whose stated condition is that “we have clear evaluation criteria, and … iterative refinement provides measurable value”.

Critic loop

1. 1Generator produces a candidate answerThe ordinary agent run, at its ordinary cost.
2. 2Critic reads the task, the rubric and the answerA separate call with a separate prompt. It does not need the tools and should not have them.
3. 3A checkable verifier runs first, where one existsA test suite, a schema validation, a citation check against the passages supplied. Free, exactly repeatable, and worth more than any model critique.
4. 4Generator revises, given the critiqueOne more generation, with the critique appended.
5. 5Stop after a fixed number of roundsTwo is usually the whole benefit. Without a hard limit this pattern is unbounded spending, because a critic can always find something.

A verifier decides; a critic opines. Put every check you can express as code before the one you cannot.

Pseudocode — not a real command

Cost with r rounds of critique = (1 + r) * generation + r * critique

critique ≈ S_c + Q + answer_size      (the critic does not need the transcript)

With r = 1 and a critique that is a fifth of a generation:  about 2.2x the base cost
With r = 3:  about 4.6x

That multiple is the reason to reach for a deterministic verifier first. Part 10’s question-answering pipeline rejects any answer citing a passage it was not given: that check costs nothing, cannot be talked out of its opinion, and catches the failure that matters most. A model critic is what you add for the judgements no check can express, and its own reliability should be measured with the harness in this part’s fourth lesson before you let it gate anything.

∴Research note

τ-bench’s authors introduce pass^k precisely because a single successful run is weak evidence: the metric evaluates “the reliability of agent behavior over multiple trials”, and the paper reports that even strong function-calling agents were “quite inconsistent”. A critic loop that turns one good run in three into two good runs in three is a real improvement, and you can only see it by running the suite several times. One run cannot distinguish a better system from a luckier one.

Parallel fan-out and merge: buying wall clock, not tokens

Fan-out sends the same task, or different parts of it, to several agents at once and merges the results. Anthropic’s parallelisation pattern distinguishes sectioning, where the subtasks differ, from voting, where the same task is attempted several times.

Parallel fan-out and merge

1. 1The task is split, or duplicatedSectioning gives each branch a different piece. Voting gives every branch the same piece and compares.
2. 2k branches run against the same serverOn one machine they share the engine. Wall clock improves only if the server batches them, and each concurrent sequence needs its own key-value cache.
3. 3Each branch returns a short resultReturn summaries, not transcripts. What the merge step reads is what the fan-out costs at the end.
4. 4A merge call reconciles themIts input is k summaries plus the task. Disagreement between branches is information: record it rather than averaging it away.

Fan-out multiplies token cost by k and divides wall clock by at most the concurrency your server can actually sustain.

Pseudocode — not a real command

Token cost   = k * (branch run) + (merge: S_m + Q + k * summary_size)
Wall clock   = max over branches + merge,  IF the server serves k concurrently
             = sum over branches + merge,  IF it serves them one at a time

That second line is the one people forget on local hardware. Part 9 measured what batching does: several sequences decoded together raise total throughput because the weights are read once for all of them. Part 23 then made the memory point: every concurrent sequence carries its own key-value cache, so k branches at a long context is k times the cache. Fan-out on a single machine is therefore a decision about slots and memory, not just about tokens, and past the point where the throughput curve flattens the extra branch buys nothing at all.

Voting has one further property worth naming: it costs k times as much for an improvement that is bounded by how often at least one branch is right. If your model gets a task right one time in ten, five branches will not fix it.

Handoffs and shared state: the cost is what you pass

When one agent hands to another, something has to go with it. The size of that something is the entire cost of the handoff, and there are only three choices.

What you pass	Cost	When it is right
The whole transcript	The transcript, again, through the second model	Almost never. It is the expensive option and the second agent did not need most of it.
A written summary	The summary, plus one generation to write it	The default. It also forces the first agent to say what it concluded.
A pointer to shared state	Almost nothing; the second agent fetches what it needs	Best, when there is somewhere to put the state: a file, an index, a checkpoint store.

smolagents implements the pointer-free version of this directly: a managed agent needs name and description attributes, which the documentation says “will then be embedded in the manager agent’s system prompt”, the same way a tool’s are, so the manager can call the managed agent by name. The documentation also gives the reason to keep memories separate: “why fill the memory of the code generating agent with all the content of webpages visited by the web search agent?”

LangGraph’s answer is the third row of that table. A checkpointer holds “short-term, thread-scoped memory” and a store holds “long-term, cross-thread memory”, so what travels between nodes is a state object rather than a conversation. If your system runs long enough that this distinction matters, that is the framework shape to want.

Choosing, and the cost of choosing wrong

Pattern	Multiplies tokens by	Helps when	Wasted when
Router	Slightly more than one specialist	Tasks fall into distinct kinds and the router is small	There is only one specialist, or the router is the big model
Planner and executor	About one, with a smaller quadratic term	Tasks take many turns and context is tight	Tasks take three turns
Critic loop	1 + r generations plus r critiques	There is a rubric and no deterministic check for it	A verifier could have answered the question for free
Parallel fan-out	k, plus the merge	Branches are genuinely independent and the server batches	The branches all consult the same passage
Handoff with summaries	One extra generation per handoff	Specialists have genuinely different tools	The “specialists” share a tool set and a prompt

Anthropic’s guidance is worth repeating as the closing rule, because it is the opposite of what a diagram of eight agents suggests: add complexity “only when it demonstrably improves outcomes”. The word doing the work is demonstrably, and the fourth lesson is how you demonstrate it. Every pattern here has a token cost you can compute in advance and a benefit you cannot; measure the benefit before you keep the pattern.

Budget communication and prevent shared-state races

Multiple agents can work independently only when their tasks and write boundaries permit it. A planner and executor have dependencies; two workers editing the same file can conflict; a critic reading an intermediate artefact can assess stale state. Specify ownership and handoff conditions before adding concurrency.

Use a task graph with explicit inputs, outputs and completion criteria. Pass evidence references and concise state at handoff rather than copying every transcript. Keep authoritative task status in one place and associate each result with the input revision it used. If a worker fails, decide whether to retry, use a partial result or stop the dependent work.

Compare with a single-agent baseline using total tokens, wall time, verified success and operator effort. Parallel work can reduce elapsed time while increasing overall computation, and shared local serving capacity can turn apparent parallelism into queueing. A critic is useful only if it detects errors and improves the final artefact under an independent check. Agreement among agents using similar models is not independent proof of correctness.

Recap

A single agent loop’s input cost grows with the square of the number of turns, because each turn resends everything before it, and every multi-agent pattern is either fighting that term or adding to it. A router spends one small-model call to avoid running every specialist, and is economical only when the router really is small and being wrong is merely expensive. A planner and executor splits n turns into m short contexts and cuts the quadratic term by a factor of m, at the price of resending the system prompt more often, which wins on long tasks and loses on short ones. A critic loop multiplies the base cost by roughly 1 + r generations plus r critiques, so every check expressible as code belongs in front of it. A parallel fan-out multiplies tokens by k and improves wall clock only as far as your server can batch, with each branch carrying its own key-value cache. And a handoff costs exactly what you pass through it, which is why a summary beats a transcript and a pointer to shared state beats both.

Check your understanding

Question 1. An agent takes about twenty turns per task and keeps running out of context. Which pattern most directly addresses that, and why?

• A critic loop, because it improves the answers
• A planner and executor, because m short executor contexts total n²/m rather than n², so the quadratic growth is divided by the number of steps
• Parallel fan-out, because branches run at the same time
• A router, because it picks the right specialist

Show the answer and why

Answer: A planner and executor, because m short executor contexts total n²/m rather than n², so the quadratic growth is divided by the number of steps

Running out of context is the quadratic term biting. Splitting the work into independent short contexts is the only pattern here that reduces it; the others add calls without shortening anything. A router changes which agent runs, not how long its transcript gets.

Question 2. You add a critic loop with two rounds of revision. Roughly what happens to the token cost per task, taking a critique to be about a fifth of a generation?

• It stays about the same, because the critic is small
• It roughly doubles
• It rises to about three generations plus two critiques, so a bit over three times the original
• It falls, because fewer retries are needed

Show the answer and why

Answer: It rises to about three generations plus two critiques, so a bit over three times the original

With r rounds the cost is (1 + r) generations plus r critiques. At r = 2 that is three generations and two critiques, about 3.4 times the base cost on the stated assumption. That multiple is why a deterministic verifier, which costs nothing and cannot be argued with, goes first.

Question 3. Which of these are true about parallel fan-out on a single machine? Select all that apply.

• It multiplies token cost by roughly the number of branches
• It always divides wall clock by the number of branches
• Each concurrent branch needs its own key-value cache, so memory scales with the branch count and the context each has reached
• Past the point where the throughput curve flattens, another branch adds latency and little total throughput

Show the answer and why

Answer: It multiplies token cost by roughly the number of branches, Each concurrent branch needs its own key-value cache, so memory scales with the branch count and the context each has reached, Past the point where the throughput curve flattens, another branch adds latency and little total throughput

The second is the one people assume and it is false. Wall clock improves only as far as the server actually serves the branches concurrently, which Part 9 measured and Part 23 costed; beyond the flat part of the curve you are paying for tokens and getting queueing.

Question 4. Why does this lesson insist the router be a genuinely small model rather than the same one that answers?

• Small models classify better than large ones
• Because the router call costs a full prefill on whichever model runs it: routing with the answering model adds latency and tokens while removing none, so the saving disappears
• Because large models cannot produce structured output
• Because the gateway cannot serve two aliases at once

Show the answer and why

Answer: Because the router call costs a full prefill on whichever model runs it: routing with the answering model adds latency and tokens while removing none, so the saving disappears

The router's value is entirely the specialists it prevents from running, weighted by the cost of the model that runs it. Put the big model in the router seat and you have added a call to the expensive model to save a call to the expensive model.

Question 5. Your agent succeeds on a task the first time you run it. What does that tell you about the pattern you just added?

• That the pattern works and can be kept
• Very little: a single run cannot separate a better system from a luckier one, which is why τ-bench proposes pass^k over repeated trials
• That the model is reliable at this task
• That the token cost was justified

Show the answer and why

Answer: Very little: a single run cannot separate a better system from a luckier one, which is why τ-bench proposes pass^k over repeated trials

The τ-bench paper introduces pass^k to "evaluate the reliability of agent behavior over multiple trials" and reports strong agents being "quite inconsistent". Repeat the suite before drawing a conclusion; the harness in this part's fourth lesson has a repeats option for exactly this.

Check my answersClear