Speculative Decoding: Draft Models, EAGLE and n-gram

By the end of this lesson you will be able to explain why guessing tokens and checking them can be faster than generating them, do the arithmetic that decides whether it pays on your workload, enable it on all three engines, and recognise the situations where turning it on makes things slower.

The idea

Part 5 established that decode is bandwidth-bound: to produce one token, the accelerator reads every active weight once. The arithmetic units are mostly idle while that traffic happens.

That idleness is an opportunity. Reading the weights to score one candidate token and reading them to score five candidate tokens costs almost the same, because the reading is the expensive part. It is the same asymmetry that makes batching work, applied along the time axis of a single conversation instead of across conversations.

So: get a cheap source to guess the next few tokens. Then run the real model once, over the whole guessed sequence at once, and ask it what it would have produced at each position. Keep the guesses that match what the real model would have done, and throw away everything from the first mismatch onwards. One expensive forward pass has produced several tokens instead of one.

The crucial property is that this is not an approximation. The accepted tokens are exactly the tokens the target model would have produced on its own, because they were checked against it. Speculative decoding changes the speed, not the output distribution. The EAGLE paper is explicit about “maintaining the distribution of the generated text”.

One speculative iteration

1. 1Propose k tokens cheaplyFrom a small draft model, a lookup in the prompt, or a lightweight head attached to the target model. Cost is a fraction of one target step.
2. 2One target forward pass over all k+1 positionsThe expensive model reads its weights once and scores every proposed position in parallel. This is the whole trick: k positions for the price of one.
3. 3Accept the longest matching prefixCompare position by position. Everything up to the first disagreement is kept; everything after it is discarded because it was conditioned on a token that will not exist.
4. 4Take one free token at the boundaryThe target pass already produced a token at the first mismatched position, so even a completely wrong guess still yields one token, exactly as ordinary decoding would.
5. 5Repeat from the new positionThe accepted tokens are the target model's own output. The distribution is unchanged; only the wall clock moves.

The worst case is one token per iteration, which is what you had before, plus the wasted draft cost. The best case is k+1 tokens for one pass over the weights.

Two iterations: a good guess and a poor one

Draft, iteration 1

d1d2d3d4

Target, iteration 1

one pass verifies all four

Kept, iteration 1

3 accepted + 1 boundary token

Draft, iteration 2

d1d2d3d4

Target, iteration 2

one pass

Kept, iteration 2

0 accepted + 1 boundary token

In the first iteration four tokens are drafted, the verification pass agrees with three of them, and the boundary token makes four kept in total. In the second the draft diverges immediately and only the boundary token survives, so that iteration paid the drafting cost for nothing. The average over many iterations is the acceptance rate, and it is the number the whole technique lives or dies by.

Where the guesses come from

vLLM’s speculative decoding page documents several methods, and they divide into two families.

Methods with no extra model. N-gram, also called prompt lookup, searches the text so far for a recent occurrence of the last few tokens and proposes whatever followed it. It costs almost nothing and it works startlingly well when the output repeats the input, which is exactly what happens in code editing, summarisation with quotation, structured extraction and any task where the model is copying. vLLM’s configuration for it names prompt_lookup_min and prompt_lookup_max as the match lengths. Suffix decoding is a related method the same page documents, matching against a suffix structure rather than a fixed window.

Methods with a model. A draft model is a small model from the same family and with the same tokeniser: Qwen3-1.7B drafting for Qwen3-32B is the canonical shape. It has to be small enough that several of its steps cost much less than one target step, and similar enough that it agrees with the target often. EAGLE and EAGLE3 are lighter still: rather than a whole second model, they attach a small head that predicts the target’s own internal features. The EAGLE paper’s argument is that “autoregression at the feature (second-to-top-layer) level is more straightforward than at the token level”, and that resolving the uncertainty in that prediction is what makes it work. It reports a latency speedup ratio of 2.7 to 3.5 times for LLaMA2-Chat 70B. MTP, multi-token prediction, uses heads the model was trained with, where the publisher provided them.

vLLM’s own method-selection guidance summarises the trade: model-based methods deliver “High gain” at low request rates, while n-gram and suffix methods give “Low to medium gain” but add no extra workload during peak traffic. It then adds the sentence that should govern your expectations: “Real gains depend on your model family, traffic pattern, hardware, and sampling settings.”

✦Tip

Start with n-gram. It needs no second checkpoint, no extra memory and no download, and on copy-heavy work it is the largest easy win in this part. Move to a draft model or EAGLE only when you have measured n-gram and found it wanting on your actual prompts.

The arithmetic that decides it

Two quantities decide whether speculation pays: the acceptance rate, the probability that a proposed token survives verification, and the draft cost, what one drafted token costs as a fraction of one target step.

Write the acceptance rate as α, the number of tokens drafted per iteration as k, and the draft cost per token as c. The expected number of tokens accepted from a chain of k proposals, plus the free boundary token, is (1 − α^(k+1)) / (1 − α). The cost of the iteration is one target step plus k draft steps, which is 1 + kc target steps. The ratio of those two is the speedup.

Three worked examples, at k = 4 and a draft costing 15 per cent of a target step:

• α = 0.7. Expected tokens ≈ 2.8, cost ≈ 1.6 steps. The arithmetic gives roughly 1.7 times the token rate.
• α = 0.4. Expected tokens ≈ 1.65, cost ≈ 1.6 steps. Roughly 1.03: you have added complexity for nothing.
• α = 0.3. Expected tokens ≈ 1.43, cost ≈ 1.6 steps. Roughly 0.89: speculation has made generation slower.

These are estimates from arithmetic, not measurements, and they ignore the memory the draft occupies and the batching effects below. What they establish is the shape: the payoff is highly non-linear in acceptance rate, and a mediocre draft is worse than no draft at all.

They also explain why k is not a free parameter. A longer chain multiplies the possible win and multiplies the cost of a miss, and because α applies at every position the chain’s expected yield saturates quickly. SGLang’s guidance on its three tuning parameters is to “leave all three unset to use auto-tuning, or set all three explicitly when tuning”, which is a sensible default position: let the engine search, and only take over when you are measuring.

∴Research note

Acceptance rate is measurable, not theoretical. vLLM exposes vllm:spec_decode_num_accepted_tokens_per_pos among its Prometheus metrics, so you can watch how many of the k positions survive in practice on your own traffic. Turn speculation on, watch that metric for a representative hour, and put the arithmetic above around it. If the acceptance rate is low, the answer is a better draft or no draft, not a longer chain.

Enabling it

vLLM takes a single --speculative-config option carrying a JSON object. The documented shapes include a draft model, {"method": "draft_model", "model": "<draft-model>", "num_speculative_tokens": 5}; n-gram, {"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_min": 2, "prompt_lookup_max": 5}; suffix decoding, {"method": "suffix", "num_speculative_tokens": 8, "suffix_decoding_max_tree_depth": 24}; and EAGLE3 with "method": "eagle3" naming the eagle head model. Acceptance is reported through the metric above.

SGLang spreads it across four options. The documentation’s EAGLE3 example is --speculative-algorithm EAGLE3 with --speculative-draft-model-path, --speculative-num-steps, --speculative-eagle-topk and --speculative-num-draft-tokens. Its algorithm list as read on 2026-09-09 includes EAGLE, EAGLE3, NEXTN, MTP, STANDALONE for a small draft model, and NGRAM. It describes --speculative-eagle-topk as the “branching factor per step”, which “improves candidate diversity and acceptance rate, but increases memory/compute consumption”.

llama.cpp does it with a draft model on the same server. The README as read on 2026-09-09 documents --spec-draft-model with the short form -md, --spec-draft-n-max with a default of 3, --spec-draft-n-min with a default of 0, --spec-draft-p-min for a minimum draft probability, and -ngld for how many draft layers go on the accelerator.

⌚Version note

None of the options in this section are in this course’s captured command reference yet, so they are named in prose rather than used in runnable blocks. Two of them are worth flagging as drift rather than as omission: the reference’s entry for llama.cpp v0.4.0 · verified 2026-09-08 records the older --model-draft and --draft spellings, while the server README as read on 2026-09-09 documents --spec-draft-model and --spec-draft-n-max. Check --help on your own build before using either. The reference is regenerated from captured help output on the validation machines, after which these options appear in runnable blocks here and on the command reference page.

vLLM’s speculative decoding page also carries the note “Pipeline parallelism is not composable with speculative decoding as of vllm<=0.15.0”. That version string is older than vLLM 0.28.0 · verified 2026-09-08, so the note may simply not have been revised; treat the combination as unverified rather than as either supported or refused, and test it if you need it.

What it costs in memory

The speedup arithmetic ignores a cost that a machine with tight memory cannot ignore.

A draft model is a second set of weights resident alongside the first. Qwen3-1.7B at four bits is around a gigabyte, which comes out of the same pool the KV cache is allocated from. On the 16 GB machine from the first lesson, that gigabyte is roughly seven thousand tokens of KV cache at FP16, which is nearly two more concurrent conversations at 4,096 tokens each. Speculation with a draft model is therefore a trade of concurrency for single-stream latency, and the previous lesson gave you the division that prices it.

An EAGLE-style head is much cheaper than a whole draft model, because it is a small head over the target’s own features rather than a second network. The n-gram and suffix methods cost nothing at all in weights: they read the text that is already there.

The draft also needs its own KV cache while it generates its chain, and the verification pass has to hold activations for k+1 positions instead of one. Neither is large next to the weights, but both are real, and both are why a machine that was exactly full before you enabled speculation may not start afterwards.

Measuring the break-even on your own machine

Everything above is arithmetic. The measurement is straightforward with the load generator from this part’s lab, and it is worth doing once for each workload you care about rather than trusting a published figure.

Run the same sweep twice, with speculation off and on, on the same model at the same context length, and compare time per output token at concurrency one, where speculation should help most, and at concurrency ten or twenty, where it may not help at all. Watch the acceptance metric while it runs. Then change the workload: a coding prompt where the model quotes back a function, a summarisation prompt over a pasted document, and a piece of open-ended writing. The same configuration will behave differently on all three, and that difference is the whole answer to “should I turn this on”.

Record the acceptance rate beside each result. It is the number that makes a speculative result reproducible, in the same way that the build tag makes a llama.cpp benchmark reproducible: without it, a tokens-per-second figure from a speculative run tells the reader nothing they can act on.

When it makes things worse

When the batch is full. This is the important one and it is the opposite of most people’s intuition. Speculation spends spare arithmetic capacity to save memory traffic. At concurrency of one, that capacity is sitting idle and the trade is excellent. At concurrency of twenty, continuous batching has already filled it, and the verification pass over k positions for every sequence is real extra work competing with real requests. This is exactly why vLLM’s method table separates gain at low request rates from behaviour “during peak traffic”, and why the same setting can help your interactive chat and hurt your evaluation sweep.

When the acceptance rate is low. Creative writing at a high temperature, a draft from a different model family, a different tokeniser, a domain the draft has never seen: all of them push α down, and the arithmetic above turns unfavourable quickly.

When the draft is too big. A draft that costs a third of a target step needs a very high acceptance rate to earn its place. Smaller and dumber usually beats larger and better, because the denominator matters as much as the numerator.

When memory is the constraint. A draft model occupies memory that would otherwise be KV cache. On a machine where concurrency is already limited by the block pool, spending gigabytes on a draft to make single-stream latency better is a poor trade if what you needed was throughput.

When it is not the bottleneck. If your users are waiting on time to first token because prompts are long, speculation does nothing: it accelerates decode, and prefill is where the time is going.

Keep distributional correctness separate from matching text

An exact speculative sampling method uses an acceptance rule and a correction distribution so the generated sequence follows the target distribution under the method’s assumptions. Simply accepting every draft token that looks plausible is a different algorithm. Greedy verification and stochastic sampling also require different reasoning about what “matches” means.

For an experiment, record the engine’s actual speculative method, the draft checkpoint or lookup configuration, the target and their compatibility requirements. Compare target-only and speculative runs on the same prompts, including a domain unlike the draft’s training examples. Save draft acceptance and end-to-end latency where available.

A low acceptance rate explains wasted draft work; a high rate alone does not establish a speed gain because drafting and verification still cost time and memory. Under high concurrency, the target may already use its compute effectively. The practical acceptance criterion is reduced latency or increased useful throughput at unchanged quality requirements. Do not require byte-identical sampled answers as proof of distributional equivalence, or infer equivalence merely because a few answers look alike.

Recap

Speculative decoding exploits the same asymmetry as batching, along the time axis: one pass over the weights can verify several proposed tokens almost as cheaply as producing one. It is exact, not approximate, because every accepted token is checked against the target model, so the output distribution is unchanged. Guesses come either from no extra model at all, with n-gram and suffix methods that shine on copy-heavy work, or from a small draft model or an EAGLE-style feature head. Whether it pays is arithmetic in the acceptance rate, the chain length and the draft’s cost, and the payoff is sharply non-linear: a draft accepted seven times in ten is worth having and one accepted three times in ten makes generation slower. vLLM configures it with a single JSON --speculative-config, SGLang with four options and an auto-tuning default, llama.cpp with a draft model on the same server. And it helps least exactly where this part started, under heavy concurrency, because there is no idle arithmetic left to spend.

Check your understanding

Question 1. Why does accepting a drafted token not change the model's output?

• Because the draft model is trained on the same data
• Because every proposed token is checked against what the target model would have produced at that position, and anything that disagrees is discarded
• Because the temperature is set to zero during verification
• It does change the output slightly, which is the cost of the speedup

Show the answer and why

Answer: Because every proposed token is checked against what the target model would have produced at that position, and anything that disagrees is discarded

Verification is the whole mechanism. The draft only proposes; the target model decides. That is why the EAGLE paper can claim it maintains the distribution of the generated text, and why speculative decoding is a latency technique rather than a quality trade.

Question 2. Your acceptance rate is about three in ten with four tokens drafted per iteration and a draft costing about 15 per cent of a target step. What should you do?

• Draft more tokens per iteration to increase the chance of a long match
• Turn speculation off or find a better draft: the arithmetic gives a ratio below one, so the drafting cost exceeds the tokens it saves
• Raise the temperature
• Increase the batch size

Show the answer and why

Answer: Turn speculation off or find a better draft: the arithmetic gives a ratio below one, so the drafting cost exceeds the tokens it saves

A longer chain multiplies both the possible win and the cost of a miss, and because acceptance applies at every position the expected yield saturates. At a low acceptance rate the extra drafting is pure loss. The fix is a draft that agrees more often, or none.

Question 3. Speculative decoding gave a clear improvement on your interactive chat and made your twenty-concurrent evaluation sweep slower. Why?

• The evaluation prompts are longer
• Speculation spends idle arithmetic capacity to save memory traffic; continuous batching has already consumed that capacity at high concurrency, so the verification work competes with real requests
• The draft model ran out of memory
• Evaluation harnesses disable speculation automatically

Show the answer and why

Answer: Speculation spends idle arithmetic capacity to save memory traffic; continuous batching has already consumed that capacity at high concurrency, so the verification work competes with real requests

Both techniques monetise the same idle arithmetic, one across conversations and one along the time axis of a single conversation. They compete. This is why vLLM's method table distinguishes gain at low request rates from behaviour during peak traffic.

Question 4. Which drafting methods need no second checkpoint? Select all that apply.

• N-gram, also called prompt lookup, which proposes whatever followed a recent match of the last few tokens
• Suffix decoding, which matches against a suffix structure built from the text so far
• EAGLE3, which attaches a head predicting the target model's internal features
• A draft model from the same family with the same tokeniser

Show the answer and why

Answer: N-gram, also called prompt lookup, which proposes whatever followed a recent match of the last few tokens, Suffix decoding, which matches against a suffix structure built from the text so far

The lookup-based methods draw their proposals from the text already present, so they cost almost nothing and need nothing downloaded. They are the right first experiment, and they are strongest exactly where the output copies the input: code editing, extraction and summarisation with quotation.

Check my answersClear