Skip to content
Local LLM Course

Faster Inference: Speculative Decoding, Caching and Draft Models

Everything in Level 3 has been about changing the model: fine-tuning it, aligning it, distilling it into something smaller. This part changes nothing about the model at all. The weights are the same, the answers are the same, and the machine is the same. Only the wall clock moves.

That is the unusual property of the two techniques here. Speculative decoding guesses several tokens cheaply and checks them all in one pass over the expensive weights, keeping only the ones the target model would have produced anyway. Prefix caching declines to recompute a prompt the server has already read. Neither is an approximation, and both can be verified: you can run the same prompts through both configurations and compare the output byte for byte. If the bytes differ, something is wrong, and this part shows you how to notice.

Part 9 introduced speculative decoding as a serving option and showed how to switch it on. This part treats it as an engineering problem instead. Where does the speed-up come from, arithmetically? What acceptance rate does a configuration need before it stops being a liability? Why does the thing that helps your interactive chat hurt your evaluation sweep? And then the part that makes this a Level 3 topic rather than a Level 2 one: the training skills from Parts 11 to 14 let you build a draft for your own fine-tuned model, which nobody else can publish for you.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • state why a verified draft token cannot change the output distribution, and describe the acceptance rule that makes sampling as exact as greedy decoding;
  • write the speed-up as a function of acceptance rate, chain length and draft cost, find the break-even point for your own configuration, and explain why the payoff collapses under concurrency;
  • measure acceptance and token rate on your own track, with and without a draft, and confirm that the completions are unchanged;
  • enable and measure prefix caching on llama.cpp, vLLM and SGLang, save and restore a conversation’s cache, and name the flags that quantise the key-value cache;
  • describe how Medusa heads and EAGLE-style drafts differ, what data each needs, and what training one costs in memory and time on your machine;
  • train a draft for your own model, serve it, and prove it is both faster and identical.

What you need first

Section titled “What you need first”

The serving work from Part 9, because this part measures servers rather than introducing them, and the gateway from that part is where the lab collects its training data. The training toolchain from Part 11 and a fine-tune of your own from Part 13, or the reference 8B model if you would rather not use your own. The lab notebook from Part 1, which every measurement here appends to.

The lab has a memory floor of 24 GB because it holds a target model, a draft model and a key-value cache at the same time, and trains a small head besides. Tracks S and N are the primary path. Track X can run everything except the training step, which its documentation does not cover; Track M has a reduced path that needs no training at all and still demonstrates the effect. Every page says which is which rather than leaving a track silent.

How to work through it

Section titled “How to work through it”

Read the three lessons in order. The first is the arithmetic, and the second and third both lean on it. Then do the lab with two terminals open, because the whole method is running two servers side by side and asking them the same questions.

One habit to carry out of this part: never quote a speculative speed-up without the acceptance rate beside it. A token rate from a speculative run is not reproducible on its own, in the same way that a llama.cpp benchmark is not reproducible without its build tag.

Part17 of 27Level3 — Model BuilderPages4Estimated time2.5 hours

0 / 4 lessons in this part completed