SGLang: When to Choose It

By the end of this lesson you will be able to say what SGLang is for, install and launch it where it is supported, name the one architectural idea that distinguishes it, and design a comparison against vLLM that produces a defensible answer rather than a preference.

Two engines, one job description

SGLang and vLLM solve the same problem and look similar from the outside. Both are Python packages with compiled kernels, both serve an OpenAI-compatible HTTP API, both do continuous batching, both manage the KV cache in blocks, both do structured output and tool-call parsing, both do tensor parallel and speculative decoding. A client that works against one works against the other.

The difference is where each project started. vLLM began from the memory-management paper you met in the first lesson: how do you stop the KV cache from wasting itself. SGLang began from a different question, which its paper states in its title: how do you efficiently execute structured language model programs, meaning workloads made of “multiple generation calls, advanced prompting techniques, control flow, and structured inputs/outputs”.

That is not an abstract distinction. It describes agents, evaluation harnesses, few-shot pipelines and anything that sends many prompts sharing large chunks of text. The paper names its two runtime contributions as “RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding”. The paper reports up to 6.4 times the throughput of the systems it was compared against in 2024, on exactly those workloads.

RadixAttention

Prefix caching in the previous lessons matched a new prompt against blocks left over from previous requests. RadixAttention organises those retained prefixes into a radix tree, so the structure of the cache mirrors the structure of the prompts that produced it.

A request meeting a radix tree of retained prefixes

1. 1The prompt arrives as a token sequenceSystem prompt, then a tool schema, then a few-shot block, then this user's question.
2. 2Walk the tree from the rootEach edge is a run of tokens already computed and still cached. The walk follows the longest path that matches this prompt.
3. 3Match ends part-way downSay the system prompt and the tool schema matched, and the few-shot block diverged. Those matched blocks are reused as they are, with no computation at all.
4. 4Prefill only the remainderThe engine computes keys and values for the tokens after the match, which may be a small fraction of the prompt.
5. 5Insert the new branch and decodeThe tokens just computed become a new branch, available to the next request that shares them. Eviction is by least recent use when the pool is full.

A linear cache can only match one history at a time. A tree lets many requests that share a long head but diverge at different points each reuse the longest prefix that applies to them, which is the shape an agent or an evaluation sweep actually produces.

--disable-radix-cache turns it off, which is worth knowing mainly as an experiment: run a prefix-heavy workload with and without it and the difference is the feature’s whole value on your workload.

Installing and launching

SGLang’s installation page gives Python 3.10 or higher, CUDA 13 by default with CUDA 12 available through an explicit torch install, and a straightforward install:

RunnableTrack N · NVIDIA GPU

SGLang into its own environment

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install --prerelease=allow sglang

The Docker image is lmsysorg/sglang:latest, with lmsysorg/sglang:latest-runtime offered for production use. As with vLLM, give it its own environment; two serving engines pinning two PyTorch builds into one environment is a fight you will lose.

The server is a Python module rather than a console script, and the documentation’s own example is:

RunnableTrack N · NVIDIA GPU

launch the SGLang server

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --host 127.0.0.1 \
    --port 30000 \
    --context-length 8192 \
    --mem-fraction-static 0.85 \
    --served-model-name local-chat

Two defaults to notice. The port is 30000, not 8000: the server arguments page gives --port a default of 30000 and --host a default of 127.0.0.1. And the memory control is --mem-fraction-static, described as the “fraction of memory for static allocation (weights and KV cache pool)” with the advice to “use smaller value for out-of-memory errors” and a computed default around 0.88. It plays the role --gpu-memory-utilization plays in vLLM, and the arithmetic behind it is identical.

--context-length is SGLang’s --max-model-len, defaulting to the model’s own config. --quantization accepts a long list including awq, fp8, gptq, gptq_marlin, awq_marlin, gguf, modelopt_fp4 and compressed-tensors, so the format discussion from the previous lesson carries over. --tp-size is tensor parallel. --enable-torch-compile is documented as accelerating “small models on small batch sizes”.

⌚Version note

The SGLang server arguments page documents several hundred options, and this course’s command reference for SGLang 0.5.19 · verified 2026-09-08 currently records fourteen of them. The runnable blocks on this page therefore use only recorded options. --grammar-backend, --tool-call-parser, --reasoning-parser, --attention-backend, --disable-radix-cache, --chunked-prefill-size, --max-running-requests, --enable-metrics and the four speculative options are confirmed in the documentation as read on 2026-09-09 and are named in prose here; the command reference is extended once they are captured from the binaries on the validation machines.

The half that is in the name

SGLang is two things, and this course uses only one of them. The paper describes a system with “a frontend language and a runtime”, where “the frontend simplifies programming with primitives for generation and parallelism control” and the runtime is what you launch with sglang.launch_server.

The frontend is a Python domain-specific language for writing programs made of many model calls: branching, running several generations in parallel, filling slots in a template, forcing a choice from a set. Its point is that when the runtime knows the structure of the program, it can schedule and cache far better than it can when it sees an unrelated stream of HTTP requests. A few-shot pipeline written in the frontend tells the runtime that all five branches share the same preamble; the same pipeline written as five independent chat completions does not, and the runtime has to rediscover it from the token prefixes.

This course serves models over HTTP and writes its clients in ordinary Python, because that is what transfers to every other engine in the course and to the agent work in Level 5. Part 26 builds agent systems where the frontend’s ideas are worth revisiting. For now, know that when SGLang’s benchmark numbers look unusually good on structured workloads, some of that comes from a programming model this course is not using, and some of it comes from RadixAttention, which you get through the plain HTTP API like everybody else.

Operating it

The server arguments page documents the same operational surface vLLM has, spelled differently.

--max-running-requests caps concurrent requests, the way --max-num-seqs does in vLLM. --chunked-prefill-size is the maximum tokens per prefill chunk, “set to -1 to disable”, which is the same trade between time to first token and inter-token latency described in the vLLM serving lesson. --enable-metrics turns on “Prometheus metrics logging”, off by default, and there is a matching set of bucket options for the time-to-first-token, inter-token-latency and end-to-end latency histograms. --log-requests and --log-level control what reaches the log, with --log-level defaulting to info.

The endpoints are the OpenAI-compatible ones you already know, so every client in this course works against SGLang by changing a port. That portability is the reason the gateway project at the end of this part is worth building: the alias hides which engine answered, and swapping one for the other becomes a configuration change rather than a migration.

Structured output and tool calling

This is where SGLang’s origins show, and it is why the next lesson treats the two engines together.

Grammar backends. SGLang’s structured outputs page names three: XGrammar, the default, which “supports JSON schema, regular expression, and EBNF constraints”; Outlines, which “supports JSON schema and regular expression constraints”; and Llguidance, which supports all three. --grammar-backend selects one. A request carries exactly one of json_schema, regex or ebnf, and the documentation is explicit that “only one constraint parameter (json_schema, regex, or ebnf) can be specified for a request”. The paper’s compressed finite state machines are what make this fast rather than a per-token filter over the whole vocabulary.

Tool calling. --tool-call-parser takes a parser name matched to the model family, and the documented set as read on 2026-09-09 includes llama3 for Llama 3.1, 3.2 and 3.3; llama4; qwen for the Qwen series other than Qwen3-Coder, with qwen3_coder for that; deepseekv3, deepseekv31 and deepseekv32; mistral; glm; gpt-oss; kimi_k2; step3; apertus2509; and pythonic for models that emit function calls as Python code. The launch form the documentation gives is python3 -m sglang.launch_server --model-path <MODEL> --tool-call-parser <PARSER_NAME>.

The parser names are not the same as vLLM’s for the same models, which is the practical fact to carry: a serving configuration is not portable between the two engines even though the client API is.

Where it runs, as of 2026-09-09

✱Hardware note

Track S — DGX Spark. NVIDIA’s DGX Spark playbook collection at https://build.nvidia.com/spark lists an sglang playbook alongside its vllm, trt-llm and llama-cpp ones, which makes SGLang a documented path on this machine. Confirm the container tag it names before building anything on it.

Track N — NVIDIA desktop or laptop. The primary and best-documented path. The install page treats NVIDIA GPU platforms as the main target.

Track X — Ryzen AI Max+ 395. The AMD GPU platform page names MI300X and MI250, and CDNA3 and CDNA4 architectures. It does not mention gfx1151, Strix Halo, Ryzen AI Max or any Radeon consumer part (checked 2026-09-09), and it does not state a required ROCm version. That is a meaningful contrast with vLLM, whose GPU installation page now names “Ryzen AI MAX / AI 300 Series (gfx1151/1150)” explicitly. On Track X, try vLLM before SGLang, and keep llama-server as the certain path.

Track M — Apple silicon. The installation page’s platform list includes an Apple Metal page alongside NVIDIA, AMD, Intel and others. This course does not teach it: Track M’s serving path is llama-server from Part 6 or the MLX server from Part 8, for the same reason given in the vLLM install lesson.

Making the comparison fair

If you run both, the temptation is to launch each with its defaults, time them, and declare a winner. That measures the defaults, not the engines.

A comparison is worth something when five things are held equal.

The same checkpoint. Not “the same model”: the same files. An FP8 build against an AWQ build is a quantisation comparison wearing an engine comparison’s clothes.

The same context length. --max-model-len in vLLM and --context-length in SGLang.

The same effective memory claim. --gpu-memory-utilization and --mem-fraction-static are defined slightly differently, so set both to a value that produces a similar block pool and record the pool size each engine reports at startup, rather than assuming the numbers mean the same thing.

The same load. The same prompts, the same output lengths, the same concurrency levels, generated by the same client. This part’s lab gives you that client.

The same reporting. Output token throughput, request throughput, and time to first token and time per output token at the same percentiles. Means will mislead you.

Then report the answer with all five conditions attached, and expect it to be workload-dependent rather than universal. On a prefix-heavy agent workload the engine with the better cache reuse should win; on independent one-shot prompts the difference should be small; on a model where one engine has a tuned kernel and the other does not, the kernel decides and the architecture is irrelevant.

∴Research note

The useful output of a comparison like this is not a winner. It is a sentence of the form: “on this machine, with this checkpoint, at this concurrency, on this prompt distribution, engine A produced more completed requests within my latency objective than engine B, by this much, on this date.” That sentence is reproducible. “SGLang is faster” is not.

So when do you choose it?

Choose SGLang when your workload is many requests sharing long prefixes, which is agents, evaluation sweeps, few-shot pipelines and repeated questions over the same document; when you want EBNF grammars as a first-class request field alongside JSON schema and regex; or when a model you care about has better support there, which does happen in both directions and changes month to month.

Choose vLLM when you want the larger ecosystem and the wider set of deployment integrations; when your track is X, where vLLM names your GPU and SGLang does not; or when something else in your stack already assumes it.

Choose neither when it is one person and one model, in which case Part 6’s llama-server starts in seconds and this whole part is optional infrastructure.

Running both is also a legitimate answer, and the gateway project at the end of this part is designed for exactly that: two engines behind one endpoint, with the model name deciding which one answers.

Measure prefix reuse with a negative control

A shared system prompt is a candidate for reuse only when its serialised token prefix matches the engine’s cache rules. Similar meaning is not enough. Adding a timestamp at the beginning can invalidate an otherwise reusable prefix even though the rest of the prompt is unchanged.

Use three conditions: a cold request, a repeat with the identical prefix, and a request with a deliberately changed early token. Keep the suffix length and answer allowance comparable. If only the identical repeat improves prompt latency and the engine reports cache hits, the evidence supports reuse. If all requests get faster, warm-up or model residency may explain the effect instead.

For a framework choice, repeat this experiment with your actual agent or retrieval prompt shape. Some workloads share long prefixes; others vary at the beginning. A cache-oriented architecture is useful when your traffic exposes reuse and when its model and API support meet your needs. Describe the traffic pattern that produced the benefit so the result remains meaningful outside a single demonstration.

Recap

SGLang and vLLM do the same job with the same client API and differ in emphasis: vLLM grew from KV memory management, SGLang from executing programs made of many related model calls. RadixAttention organises retained prefixes as a tree so that requests sharing long heads reuse the longest match available, which is the shape agent and evaluation workloads produce. Installation is a pip or uv install into its own environment, or the lmsysorg/sglang image; the server is launched as a Python module, listens on port 30000 by default, and uses --mem-fraction-static where vLLM uses --gpu-memory-utilization. Structured output offers XGrammar by default with Outlines and Llguidance as alternatives, and tool calling uses per-family parser names that are not the same as vLLM’s. As of 2026-09-09 its AMD page names MI300X and MI250 and does not name gfx1151, so Track X should try vLLM first, and Track M uses llama-server or MLX. A comparison between the two engines is worth something only when the checkpoint, the context, the memory claim, the load and the reporting are all held equal, and its output is a dated sentence rather than a winner.

Check your understanding

Question 1. What does RadixAttention do that a linear prefix cache does not?

• It compresses the KV cache
• It organises retained prefixes as a tree, so many requests that share a long head but diverge at different points can each reuse the longest prefix that applies to them
• It stores the cache on disk
• It removes the need for a KV cache entirely

Show the answer and why

Answer: It organises retained prefixes as a tree, so many requests that share a long head but diverge at different points can each reuse the longest prefix that applies to them

A tree matches the shape of the workload. An agent loop sends a fixed system prompt and tool schema followed by a growing and branching history; a tree lets every branch reuse everything up to its divergence point, which a single linear history cannot.

Question 2. You launch SGLang with the documentation's example command and your client cannot connect on port 8000. Why?

• The server failed to start
• SGLang's documented default port is 30000, not 8000
• SGLang does not expose an HTTP API
• The API key is missing

Show the answer and why

Answer: SGLang's documented default port is 30000, not 8000

The server arguments page gives --port a default of 30000 and --host a default of 127.0.0.1. This is a small thing that costs people twenty minutes, and it is a good reason to set both explicitly on every launch, as this course does.

Question 3. Which of these make a vLLM-versus-SGLang comparison defensible? Select all that apply.

• The same checkpoint files, not merely the same model name
• The same context length and a comparable resulting block pool, recorded from each engine's startup output
• The same prompts, output lengths and concurrency levels from the same client
• Each engine run with its own defaults, so each is shown at its best

Show the answer and why

Answer: The same checkpoint files, not merely the same model name, The same context length and a comparable resulting block pool, recorded from each engine's startup output, The same prompts, output lengths and concurrency levels from the same client

Running each at its defaults measures the defaults. The memory options in particular are defined differently in the two engines, so setting the same number does not produce the same block pool; record what each engine actually allocated.

Question 4. What is the status of SGLang on Track X, the Ryzen AI Max+ 395, as read on 2026-09-09?

• Fully supported, with a dedicated installation path
• The AMD GPU page names MI300X, MI250 and the CDNA3 and CDNA4 architectures, and does not name gfx1151, Strix Halo or any Radeon consumer part
• Explicitly unsupported, with an error message on startup
• Supported only through the CPU backend

Show the answer and why

Answer: The AMD GPU page names MI300X, MI250 and the CDNA3 and CDNA4 architectures, and does not name gfx1151, Strix Halo or any Radeon consumer part

Absence from a supported list is not the same as a refusal, but it is not a path this course can recommend either. vLLM does name gfx1151 on its GPU installation page, which makes it the one to try first on that machine, with llama-server as the certain fallback.

Check my answersClear