Sampling: Temperature, Top-p, Min-p, Repetition and Determinism

By the end of this lesson you will be able to explain what each sampling control does to the model’s probability distribution, choose settings deliberately rather than by copying somebody’s configuration, quote what the course’s reference models ask for and say where that came from, and explain to a colleague why the same prompt with the same seed produced two different answers without anything being broken.

A distribution, then a choice

Part 2 established what a language model produces: for every position, one score per token in the vocabulary. Those scores are logits, and turning them into a probability distribution takes one function, the softmax, which exponentiates each score and divides by the total so that the numbers sum to one.

That distribution is the model’s entire output. Everything after it is the sampler, and the sampler is not part of the model. It is a small piece of ordinary code in the engine that takes a distribution and returns one token, and every setting in this lesson is a way of reshaping the distribution before the draw.

From the model's scores to one token

1. 1LogitsOne raw score per vocabulary token, straight from the last layer. Not probabilities yet.
2. 2PenaltiesAdjust the scores of tokens that have already appeared: repetition, presence, frequency.
3. 3TruncationDiscard the unlikely tail: top-k keeps a fixed number, top-p keeps a probability mass, min-p keeps everything above a fraction of the best token.
4. 4TemperatureFlatten or sharpen what is left. Below one sharpens towards the most likely token; above one flattens towards the rest.
5. 5DrawOne token, chosen at random from the surviving distribution. Append it and start again for the next position.

The order matters, because truncating then flattening is not the same as flattening then truncating. llama.cpp prints the sampling settings it will use before it generates; read that print-out rather than assuming an order from a blog post, because the chain is configurable and has changed over the project’s life.

Temperature

Temperature divides the logits before the softmax. Divide by a number below one and the gaps between scores grow, so the leading token takes more of the probability mass. Divide by a number above one and the gaps shrink, so unlikely tokens get a real chance.

At a temperature of zero the sampler stops sampling and takes the highest-scoring token every time. This is greedy decoding, and it is the setting people reach for when they want reproducibility. It has two costs: it removes the variety that makes a model useful for drafting and brainstorming, and on some models it produces loops, because the highest-probability continuation of a repeated phrase is often that phrase again. Several of the course’s reference models say so on their own cards.

Truncation: top-k, top-p and min-p

Temperature alone leaves the whole vocabulary in play, including tokens the model scored at nearly zero. With a flattening temperature that is a real risk, so samplers cut the tail first. Three ways, in the order they were invented:

Top-k keeps the k highest-scoring tokens and discards everything else. Simple, and blunt: k is a fixed number whether the model is certain or uncertain. Twenty is a common value.

Top-p, also called nucleus sampling, sorts the tokens by probability and keeps the smallest set whose probabilities add up to p. This adapts: when the model is confident, two or three tokens reach 0.95 between them, and when it is unsure the set is much larger.

Min-p keeps every token whose probability is at least a given fraction of the most likely token’s. If the best token has probability 0.6 and min-p is 0.05, the threshold is 0.03. It adapts in a different way from top-p: when one token dominates, min-p is aggressive; when the field is level, it is permissive.

You do not need all three. Most published recommendations combine top-k with top-p, and some suggest min-p as a replacement for both. What matters is knowing which are switched on, because a sampler you did not set is not a sampler that is off: it has a default.

Temperature, worked through

The effect is easier to see on four tokens than in the abstract. Suppose the model’s probabilities for the next word, after the softmax at temperature one, are as in the first column. Lowering the temperature raises the leading token’s share and flattens the rest towards zero; raising it does the opposite.

Token	At temperature 1.0	At temperature 0.5	At temperature 1.5
“ the”	0.500	0.685	0.421
“ a”	0.300	0.247	0.300
“ some”	0.150	0.062	0.189
“ every”	0.050	0.007	0.091

Those figures are worked from the arithmetic rather than measured from a model. Dividing the logits by the temperature is the same as raising the probabilities to the power of one over the temperature and renormalising, so the second column is each probability squared and rescaled, and the third is each raised to the power of two thirds and rescaled. The shape is the point. At 0.5 the leading token takes more than two thirds of the mass and the fourth is effectively gone; at 1.5 the fourth token has nearly doubled its chance while the leading one has given up a sixth of its own. Over a five-hundred-token answer, a small change at each step compounds into a noticeably different piece of writing.

Which control to reach for

Given four dials, people tend to turn all of them. A more useful habit is to know what each one is for.

The answer is repetitive or looping. Look at the template and the prompt first, then the temperature, then the penalties. Loops at temperature zero are a known behaviour of several of the course’s reference models and the fix is to stop using temperature zero, not to add a penalty.

The answer is wandering, inventing details, or changing the subject. Lower the temperature, or tighten the truncation with a smaller top-p or a larger min-p. Sampling from the tail is exactly what produces plausible-sounding invention.

The answer is dull, or every regeneration is identical. Raise the temperature, or loosen the truncation. This is what those controls are for, and it is a legitimate thing to want when drafting.

The answer must be valid JSON, or must match a schema. Do not use the sampler for this at all. The engine supports constrained decoding through GBNF grammars, and the server exposes a grammar option and a JSON-schema option; constraining the sampler to tokens that keep the output valid is strictly better than lowering the temperature and hoping. Part 22 builds on that.

The tools also offer other samplers, --mirostat among them, which target a measure of surprise rather than a fixed cut. They are worth knowing exist and are not worth adopting without measuring against the model card’s recommendation on your own task.

Penalties

Three controls act on tokens that have already appeared in the context.

Repetition penalty (--repeat-penalty) divides the score of tokens that occurred recently, with --repeat-last-n setting how far back “recently” reaches. Presence penalty subtracts a fixed amount from any token that has appeared at all. Frequency penalty subtracts an amount proportional to how often it has appeared.

They are cruder than they look. A model writing code or a numbered list needs to repeat itself, and a penalty cannot tell that repetition from a loop. Symptoms of a penalty set too high are lost formatting, invented synonyms and a drift away from the requested vocabulary. Start at the model card’s recommendation, and treat a repetition problem as evidence about the prompt or the template before you reach for a penalty.

Seeds, and why the same seed can still differ

-s sets the random seed. Same seed, same settings, same build, same machine, same batching: the same answer. Change any of those and the promise weakens.

Two things make this fragile on a GPU, and neither is a bug.

Floating-point addition is not associative. Adding a million small numbers in a different order gives a slightly different sum. GPU kernels split a reduction across many threads and combine the partial results in whatever order the hardware finished them, and the number of threads depends on the batch size and the device. So the logits differ in their last bits between runs with different batch shapes, and when two tokens are nearly tied, a difference in the last bits flips the choice. One flipped token changes everything after it.

Batching changes the shape. With continuous batching and parallel slots, your request is processed alongside whatever else arrived. A different set of neighbours means a different batch size, which means a different reduction order.

!Important

Greedy decoding is not a reproducibility mechanism either. It removes the random draw, but it does not remove the near-ties, and a near-tie decided differently by the last bit of a sum is a different answer. Reproducibility on a GPU is a property of the whole configuration, not of the seed. If you need a run you can reproduce exactly, record the build tag, the backend, every sampling setting, the batch and slot settings and the seed, and expect to re-run on the same machine.

This is why the course’s evaluation method, from the first reality check in Part 3 onwards, is to run several samples and report a distribution rather than to run one sample and treat it as the model’s answer. Part 16 builds that into a proper evaluation harness.

What the reference models ask for

These are not the course’s recommendations. They are what each publisher put on its own model card, which is the closest thing to an authority that exists, because the publisher chose them by measuring.

Qwen3, including Qwen3-8B and the other members of the family, gives two sets under “Best Practices”, one per mode:

Setting	Thinking mode	Non-thinking mode
Temperature	0.6	0.7
Top-p	0.95	0.8
Top-k	20	20
Min-p	0	0
Presence penalty	not specified	0 to 2, optional, to reduce repetition

The card adds two instructions worth repeating. In thinking mode it says, in capitals, not to use greedy decoding, because it “can lead to performance degradation and endless repetitions”. And it recommends an output length of 32,768 tokens for most queries, with 38,912 suggested for complex problems, which is a reminder that a reasoning model’s answer includes the working.

gpt-oss takes a different approach. Reading the gpt-oss-20b card on 2026-09-09, it gives no recommended temperature or top-p. What it does give is a reasoning-effort control with three levels, low, medium and high, “set in the system prompts, e.g., ‘Reasoning: high’”, and a firm requirement that the model be used with the harmony response format, because it “was trained on our harmony response format and should only be used with the harmony format as it will not work correctly otherwise”. For this model the effort level and the format are the settings that matter, and the sampler is left to you.

RunnableAll tracks

Qwen3 in thinking mode, with the card's settings

~/llama.cpp/build/bin/llama-cli \
    -m ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    -cnv -c 16384 \
    --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 \
    -s 1234

RunnableAll tracks

the same settings on the server, plus a presence penalty

~/llama.cpp/build/bin/llama-server \
    -m ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    -c 16384 -ngl 99 \
    --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 \
    --presence-penalty 1.0 --repeat-last-n 64

⌚Version note

Model cards change, and sampling advice changes with the model. The settings above were read from the Qwen3-8B card and the gpt-oss-20b card on 2026-09-09. Before you rely on them for a model this course does not cover, read that model’s own card; before you rely on them for a fine-tune, remember that the fine-tune may have changed the behaviour the numbers were chosen for.

Sampling for reasoning models

A reasoning model produces its working before its answer, usually inside a marked block. That changes three things about sampling.

The output is much longer, which is why Qwen3’s card recommends a large output length and why your -c needs to accommodate working plus answer plus the conversation so far.

Greedy decoding is discouraged rather than merely unfashionable. A long chain of steps at temperature zero has many opportunities to fall into a loop, and the Qwen3 card is explicit about the consequence.

The working is not part of the conversation. Qwen3’s card notes that the history in a multi-turn conversation “should only include the final output part and does not need to include the thinking content”. Sending the working back in the next turn wastes context and confuses the format. Most clients strip it; if you write your own, strip it yourself.

✕Common mistake

Judging a reasoning model on its working. The block before the answer is intermediate text, not a claim, and it is not a reliable account of how the answer was produced. Read it for clues when debugging, do not quote it as the model’s reasoning, and do not score it. Part 16 scores answers.

Reproducibility needs more than a seed

Separate three levels of repeatability: identical request settings, statistically similar behaviour and byte-identical output. A fixed seed addresses the random-number sequence in the implementation that honours it. It does not freeze all floating-point reductions, batching order, kernels, hardware or server scheduling.

For an experiment, send the same request several times without changing the checkpoint, template or server configuration. Save complete responses and compare both exact text and task success. Then repeat after changing one setting such as temperature. If the exact words vary while the required fields remain correct, the application may already be sufficiently stable. If the verdict changes, report a success rate rather than selecting the favourable run.

Greedy decoding removes sampling randomness by selecting an argmax, but near-tied logits can still respond to numerical differences. Treat deterministic generation as a measured property of a recorded stack. For evaluations, pin settings and repeat borderline comparisons; for applications, validate the output contract and handle failure even when your smoke test happened to repeat exactly.

Recap

The model produces logits; the sampler turns them into a token, and the sampler is engine code rather than model behaviour. Penalties adjust tokens that already appeared, truncation with top-k, top-p or min-p removes the unlikely tail, temperature sharpens or flattens what remains, and one token is drawn. A seed reproduces a run only when the build, the backend, the batch shape and every setting are also the same, because floating-point reductions on a GPU do not commit to an order, and greedy decoding does not fix that. The publishers’ cards are the place to get starting values: Qwen3 gives one set for thinking mode and another for non-thinking, and tells you not to use greedy decoding in the former; gpt-oss gives a reasoning-effort level and a required response format instead of sampler values.

Check your understanding

Question 1. What does temperature actually do?

• Selects how many tokens the model considers
• Divides the logits before the softmax, so a value below one sharpens the distribution towards the leading token and a value above one flattens it
• Sets the random seed
• Penalises tokens that have already appeared

Show the answer and why

Answer: Divides the logits before the softmax, so a value below one sharpens the distribution towards the leading token and a value above one flattens it

It is a scale factor on the scores. Truncation controls, top-k, top-p and min-p, are what decide which tokens are considered at all, and they act on a different part of the pipeline.

Question 2. The model is very confident: one token has probability 0.9. How do top-p 0.95 and min-p 0.05 differ here?

• They keep the same set
• Top-p keeps the leading token plus enough others to reach 0.95, while min-p keeps only tokens with probability at least 0.045, which is a much tighter cut when one token dominates
• Min-p keeps more tokens than top-p in every case
• Neither applies when the model is confident

Show the answer and why

Answer: Top-p keeps the leading token plus enough others to reach 0.95, while min-p keeps only tokens with probability at least 0.045, which is a much tighter cut when one token dominates

Min-p scales its threshold to the best token, so confidence makes it strict. Top-p scales to a cumulative mass, so confidence makes it small too but by a different rule. That difference in behaviour is the reason both exist.

Question 3. Two runs on the same GPU with the same seed and settings produced different answers. Which explanation fits?

• The model weights changed
• Floating-point reductions on a GPU are not committed to an order, so logits can differ in their last bits; a near-tie decided the other way changes every token after it
• The seed was ignored
• A different quantisation was loaded

Show the answer and why

Answer: Floating-point reductions on a GPU are not committed to an order, so logits can differ in their last bits; a near-tie decided the other way changes every token after it

Addition is not associative in floating point, and GPU kernels combine partial sums in whatever order the threads finish. A different batch shape, from continuous batching or a different slot count, is enough to change that order.

Question 4. What does the Qwen3 card say about greedy decoding in thinking mode?

• It is recommended for reproducibility
• Do not use it, because it can lead to performance degradation and endless repetitions
• It has no effect on reasoning models
• It should be combined with a high presence penalty

Show the answer and why

Answer: Do not use it, because it can lead to performance degradation and endless repetitions

The card states it in capitals. A long chain of steps at temperature zero has many chances to enter a loop, which is why the card gives a temperature of 0.6 with top-p 0.95 and top-k 20 for thinking mode instead.

Check my answersClear