Skip to content
Local LLM Course

Serving at Scale: vLLM and SGLang

Every engine so far has been built around one person waiting for one answer. llama.cpp in Part 6 is excellent at that, and Ollama and LM Studio in Part 7 make it comfortable. This part is about what changes when twenty requests arrive at the same time, which is the situation every remaining level of this course puts you in.

It is worth saying why that matters before you have any users. Levels 3 to 5 do not run models interactively. Evaluation harnesses send hundreds of prompts and wait for hundreds of answers. Synthetic data generation sends thousands. Reinforcement learning rollouts send a batch, score it, and send another. A coding agent sends one request, gets a tool call back, sends another, and does that fifty times before it finishes a task. In all four cases the number you care about is not how fast one answer arrives; it is how much total work the machine gets through, and whether the slowest request in the batch is still tolerable.

vLLM and SGLang are the engines built for that shape of work. They are not faster versions of llama.cpp. They make a different trade: more memory reserved up front, a heavier install, no Metal backend, in exchange for a scheduler that keeps the accelerator busy across many requests and a KV cache that is managed like operating-system memory rather than like one big array per conversation.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • explain what continuous batching and paged attention do, why the second concurrent request is much cheaper than the first, and when llama.cpp remains the better choice;
  • install vLLM on your track, or say precisely why your track cannot run it and which engine takes its place;
  • serve a quantised model with vLLM, choose the context length and memory fraction deliberately, spread a model across two GPUs, and read the server’s own logs and metrics rather than guessing;
  • say what SGLang does differently, install it where it is supported, and make a fair comparison against vLLM on the same model;
  • make a local server produce reliable function calls and schema-valid JSON, which is the property Part 24 and Part 25 depend on entirely;
  • decide whether speculative decoding is worth enabling for your workload, and recognise the case where it makes generation slower;
  • load-test a served model at four concurrency levels, produce a throughput-against-latency curve for your own machine, and show the effect of prefix caching on repeated prompts;
  • run a gateway that puts one endpoint, one key and a set of stable model names in front of every engine on your machine, and document it well enough that later parts can just use it.

What you need first

Section titled “What you need first”

Part 6, because llama-server is the comparison in every measurement here and the fallback on any track vLLM cannot serve. Part 5’s memory arithmetic, because the whole of the vLLM serving lesson is an argument about where memory goes. Part 4’s model library in ~/models, because the gateway loads GGUF files straight out of it.

You also need a container runtime on Tracks S, X and N. Two of the four install paths in this part are container paths, and the gateway project ships a Compose file. Track M runs the gateway natively instead, with scripts provided.

The memory floor is 16 GB for the load-testing lab, because twenty concurrent slots need KV cache that a single-user setup does not, and 8 GB for the gateway project, which loads one small model at a time.

How to work through it

Section titled “How to work through it”

The six lessons build on each other and the two hands-on pages at the end use all of them. Read the first lesson away from the machine; it is the concepts, and everything afterwards is easier once continuous batching and paged attention are clear. Do the install lesson with a terminal open, and stop at its verification step until a model actually answers.

The serving lesson is the long one, and it is worth working through with a server running so you can watch each option change the startup log. The SGLang lesson can be read; whether you install it depends on your track and on whether the comparison interests you. The tool-calling lesson is the one that Level 5 leans on hardest, so do not skim it because it looks like plumbing.

Then give the lab an uninterrupted hour. It produces the first numbers in this course that describe your machine under load rather than under one user, and Part 22 and Part 23 refer back to them.

Finish with the gateway project. It is deliberately placed at the end of Level 2 because everything after it assumes a stable endpoint and a stable set of model names. Write the GATEWAY.md file properly: you will be reading it in Part 13, Part 25 and the capstone.

Part9 of 27Level2 — Local OperatorPages8Estimated time5.2 hours

0 / 8 lessons in this part completed