Prompting That Works Locally: System Prompts, Chat Templates and Thinking Modes

A model that answers badly is usually being asked badly, and on a local machine “badly” often means “in a format it was never trained on”. By the end of this lesson you will be able to say exactly what reaches the model when you send a chat message, spot the three formatting faults that make a competent model look incompetent, switch a model’s reasoning mode deliberately in both directions, and set the sampling parameters its own publisher recommends instead of the ones your engine happened to default to.

Your messages are not what the model receives

The Transformers documentation puts the mechanism plainly: “All causal LMs, whether chat-trained or not, continue a sequence of tokens.” There is no messages array inside the model. Somewhere between your API call and the forward pass, a list of role-and-content dictionaries is flattened into one string of tokens, and the model continues it.

The flattening is done by the model’s chat template, a small program shipped with the tokeniser. Different families flatten differently. Mistral-7B-Instruct wraps user turns in [INST] and [/INST]; Zephyr uses <|user|> and <|assistant|>; the Qwen3 family uses the ChatML tokens <|im_start|> and <|im_end|>. The documentation’s warning about mixing them up is not subtle: “with the wrong control tokens, these models would have drastically worse performance.”

What happens to a chat message on the way to the weights

your request → tokens

1. Your requestA list of {role, content} objects, plus sampling parameters.you write this
2. Chat templateThe Jinja program in the model's tokeniser config, which turns the list into one string with the family's control tokens.ships with the model
3. Generation promptThe tokens that say "an assistant turn starts here". Without them the model may continue your message instead of replying to it.
4. TokeniserThe string becomes token ids. Special tokens added twice are a common and silent fault.
5. EnginePrefill over those ids, then decode, under the sampler you configured.you configure this

Two of these five layers ship with the model and two are yours. Most prompting problems on a local machine are in the middle layer, which is the one nobody looks at.

Three things go wrong here often enough to be worth naming.

The wrong template. A GGUF file carries the template in its metadata, and llama-server uses it when started with --jinja, which its README documents as “whether to use jinja template engine for chat (default: enabled)”. A file converted or re-packaged carelessly can carry a generic template instead of the model’s own, and the symptom is a model that rambles, ignores the system prompt, or never stops. --chat-template and --chat-template-file exist to override it when you have a better copy.

The missing generation prompt. apply_chat_template(..., add_generation_prompt=True) appends the tokens that open an assistant turn. The documentation is explicit about what happens without them: “the model may get confused and do something strange, like continuing the user’s message instead of replying to it”. If you use an OpenAI-compatible chat endpoint this is handled for you; if you build prompts by hand for the completion endpoint, it is yours to get right.

Double special tokens. The template already inserts the beginning-of-sequence token. The documentation warns that “adding additional special tokens is often incorrect or duplicated, hurting model performance”, and recommends add_special_tokens=False if you format with tokenize=False and tokenise later.

✕Common mistake

A model that answers the previous question, repeats your prompt back, or emits <|im_start|> in its reply is telling you about a template fault, not about its ability. Before you change models, ask the server for the prompt it actually built: enable verbose logging on llama-server and read the first request in the log. Ten seconds there saves an afternoon of prompt engineering against the wrong problem.

Seeing the prompt the server actually built

You can stop guessing about all three. Two habits are enough.

The first is to render the template yourself, outside the engine, for any model whose behaviour surprises you. apply_chat_template with tokenize=False returns the exact string the model will be given, and reading it takes ten seconds. You are looking for three things: the family’s control tokens where you expect them, an assistant turn opened at the end, and no doubled beginning-of-sequence token.

Fragment — not complete on its own

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
messages = [
    {"role": "system", "content": "Answer in one sentence."},
    {"role": "user", "content": "Why is decode bandwidth-bound?"},
]
print(tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))

The second is to make the server tell you. Started with verbose logging, llama-server writes what it received and what it built, and comparing that against the string above settles the question of whether the engine is using the template you think it is. Do this once for each new model you add to a service, not every time something looks odd; it takes a minute and it removes a whole category of mystery.

✦Tip

When you convert or download a GGUF from a third party, check the template before you check anything else. A re-packaged file with a generic template is common, produces output that is subtly worse rather than obviously broken, and is indistinguishable from “this model is not very good” unless you look.

System prompts: the part of the prompt you pay for every time

The system role holds directives about how the model should behave, and templates put it at the front. That position has two consequences.

The first is that it is read on every request. A three-hundred-token system prompt is three hundred tokens of prefill added to every question anybody asks, forever. That is cheap if the prefix is cached and it is not cheap otherwise, which is the subject of the last section of this lesson.

The second is that everything after it is conditioned on it. This is what makes system prompts worth writing carefully and worth keeping short. Put the invariants there: the role, the output format, the refusal rule, the language. Put the task in the user turn.

Smaller models tend to hold fewer simultaneous constraints than larger ones, but “tend to” is not a measurement, and the number of instructions your model will actually follow is something you can find out rather than guess. That is exactly what the lab at the end of this part is for: write the same task with three, six and ten constraints, and score the results.

Thinking modes, and how to drive them

Several current open-weight families can produce a visible reasoning trace before their answer, and the switch is part of the prompt rather than a separate model.

Qwen3 generates its reasoning inside a <think> block. The card describes enable_thinking=True, the default in apply_chat_template, as the setting where “the model will generate think content wrapped in a <think>...</think> block”, and enable_thinking=False as the setting where it “will not generate any think content and will not include a <think>...</think> block”. On top of the hard switch there are soft switches: with thinking enabled you can add /think or /no_think to a message, and the card states that “the model will follow the most recent instruction in multi-turn conversations”. The trap is that these are only soft switches: when enable_thinking=False “the soft switches are not valid”, so a /think in your text does nothing and you will spend twenty minutes wondering why.

gpt-oss exposes an effort level instead of a binary. The card documents setting Reasoning: low, Reasoning: medium or Reasoning: high in the system prompt, described respectively as “fast responses for general dialogue”, “balanced speed and detail” and “deep and detailed analysis”. It also states that the models “were trained on our harmony response format and should only be used with the harmony format as it will not work correctly otherwise”, which is the same chat-template point in a stronger form.

Qwen3-Coder has no thinking mode at all. Its card says the model “supports only non-thinking mode and does not generate <think></think> blocks in its output”. Asking it to think step by step is fine as ordinary prompting; expecting a reasoning block is not.

▤Memory budget

Reasoning tokens are ordinary tokens. They are decoded one at a time at the same rate as answer tokens, they occupy the same key-value cache, and they count against the context length you allocated in Part 6. A model asked to think for two thousand tokens before a fifty-token answer spends roughly forty times as long producing the part you will not read as the part you will. That is often worth it. It is never free, and the arithmetic in Part 5 is how you price it before you enable it for every request in a service.

The settings the publisher actually recommends

Engines ship with generic sampler defaults. Model cards ship with specific ones, and the gap between them is a common and invisible cause of poor output.

Qwen3’s card recommends, for thinking mode, Temperature=0.6, TopP=0.95, TopK=20 and MinP=0, with an instruction in capital letters not to use greedy decoding, “as it can lead to performance degradation and endless repetitions”. For non-thinking mode it recommends Temperature=0.7, TopP=0.8, TopK=20 and MinP=0. It suggests a presence penalty between 0 and 2 to reduce repetition, while noting that a higher value may cause occasional language mixing and a slight drop in quality. It also recommends an output length of 32,768 tokens for most queries and 38,912 for complex problems, which is a budget, not a target. Qwen3-Coder’s card recommends temperature 0.7, top-p 0.8, top-k 20 and a repetition penalty of 1.05.

Those are four different numbers for two models from the same publisher, which is the point: read the card for the model you are running.

RunnableAll tracks

start a server with the card's non-thinking settings

llama-server \
  --model ~/models/qwen3-8b/Qwen3-8B-Q4_K_M.gguf \
  --alias qwen3-8b \
  --jinja \
  --ctx-size 8192 \
  --temp 0.7 \
  --top-p 0.8 \
  --top-k 20 \
  --min-p 0 \
  --host 127.0.0.1 \
  --port 8080

⌚Version note

Sampler defaults change between releases, and a value you set on the command line can still be overridden per request by the client. Record the settings that produced any result you intend to compare against later; the evaluation harness in this part’s lab records them for you, which is most of why it exists. Checked against the model cards on 2026-09-08.

Few-shot examples, priced

Showing the model two or three worked examples is the most reliable way to fix output format without touching the model. It is also the most reliable way to make every request slower, because the examples are prefill.

The arithmetic is easy and worth doing out loud. Eight examples of about two hundred tokens each is sixteen hundred tokens of prefix on every single request. Prefill is compute-bound, as Part 3 established, so on most machines that is a fixed cost added to the time before the first token appears. Whether it matters depends on whether that prefix is read again from scratch every time.

Where the tokens in one request go

1. 1System promptInvariant. Same on every request.
2. 2Few-shot examplesInvariant. Same on every request.
3. 3Retrieved or pasted contextVaries per request. Put it after the invariant part.
4. 4The questionShort. Varies per request.
5. 5The answerDecoded one token at a time.

Only the last box is affected by how long your answer is. The first three are paid on every request unless the prefix is cached.

There is a point at which few-shot prompting stops being the right tool. If you need twenty examples to get the format right, you want either constrained decoding, which the next lesson covers and which makes the format structurally impossible to break, or a fine-tune, which Part 13 covers and which moves the examples out of the prompt and into the weights.

Prefix reuse: the reason the order of your prompt matters

Because a transformer’s key-value cache is built left to right, the cached state for a prefix is still valid as long as the prefix has not changed. llama-server documents a prompt cache that is enabled by default, an option to set the minimum chunk size it will attempt to reuse from that cache by shifting the key-value entries, and a slot-save path for persisting a slot’s cache to disk. Part 9’s engines generalise this: prefix caching shared across requests is one of the main reasons a production server beats a single-user one when many people ask similar questions.

The practical rule follows from the mechanism. Everything invariant goes first. System prompt, then examples, then documents, then the question. Change one character in the system prompt and every cached token after it is worthless. Put a timestamp at the top of your prompt and you have disabled prefix caching for your whole service without noticing.

✦Tip

Test this rather than believe it. Send the same long prefix twice with different final questions and compare the time to first token on the two requests. If the second is not noticeably faster, the cache is not being used, and the usual reason is that something in your prefix, a timestamp, a request id, a shuffled example order, is changing between requests.

Prefilling the answer

One more lever, useful and under-used. continue_final_message in apply_chat_template removes the end-of-sequence tokens from a trailing assistant message so the model continues it instead of starting a new one. The documentation describes it as “prefilling” a model response and notes it “can be very useful for improving the accuracy of instruction following when you know how to start its replies”. Ending the conversation with an assistant message containing {"name": " is a blunt but effective way to get JSON. The documentation also warns that add_generation_prompt and continue_final_message cannot be used together and returns an error if you try.

For reasoning models there is a subtlety worth knowing: prefilling content closes the reasoning block before generation starts, so a prefill has to target the reasoning field itself if you want the model to continue thinking rather than answering.

Run a prompt ablation you can interpret

Choose one fixed task set and compare a plain instruction, the same instruction with a schema, and the schema plus a few examples. Keep the model, template, sampling and output budget unchanged. Save rendered prompts if the server exposes them, so an improvement can be attributed to the intended prompt change.

Examples should demonstrate edge cases as well as the easy path. For extraction, show an absent value and conflicting evidence; otherwise the model may learn from your prompt that every field should be filled. Keep untrusted document text clearly separated from application instructions, while enforcing permissions outside the prompt.

Report per-category changes and total tokens. A longer prompt that raises quality but doubles latency may be appropriate for offline analysis and poor for interactive completion. If a reasoning mode is changed at the same time, it is a separate treatment because it can change output length and computation. Prompting is an experiment with a measurable task contract, not a collection of phrases that improve every model.

Recap

The model receives one token string, not your messages, and the chat template is what produces it. The wrong template, a missing generation prompt or a duplicated special token each degrade a good model, and each is invisible unless you look at the prompt the server built. System prompts are read on every request, so keep them invariant and put them first. Thinking modes are prompt-level switches with per-family syntax: Qwen3’s enable_thinking plus /think and /no_think, gpt-oss’s reasoning levels in the system prompt, and no thinking mode at all in Qwen3-Coder. Use the sampling settings the model card recommends, record them with every result, and remember that reasoning tokens cost decode time and context. Few-shot examples fix format at a fixed prefill cost, prefix caching makes that cost survivable, and both depend on putting the invariant part of the prompt first.

Check your understanding

Question 1. A model starts replying and then emits <|im_start|>assistant in the middle of its answer. What is the first thing to check?

• The temperature is too high
• The chat template: the wrong one is being applied, or special tokens are being added twice on top of the template
• The context length is too small
• The model file is corrupt

Show the answer and why

Answer: The chat template: the wrong one is being applied, or special tokens are being added twice on top of the template

Control tokens leaking into visible output is a formatting fault, not a sampling one. Either a generic template was applied instead of the family's own, or the string was tokenised again with special tokens enabled after the template had already inserted them.

Question 2. You set enable_thinking=False for a Qwen3 model and then send a message containing /think. What happens?

• The model produces a <think> block for that turn
• Nothing: the card states that when enable_thinking=False the soft switches are not valid
• The server returns an error
• The model thinks for every subsequent turn as well

Show the answer and why

Answer: Nothing: the card states that when enable_thinking=False the soft switches are not valid

The hard switch removes the thinking machinery from the rendered prompt entirely, so there is nothing for the soft switch to turn on. The soft switches only work in the mode where thinking is already available.

Question 3. Which prompt layout makes prefix caching most effective?

• Question first, then examples, then the system prompt
• A timestamp, then the system prompt, then examples, then the question
• System prompt, then examples, then retrieved documents, then the question
• The order makes no difference because the cache is keyed on the whole prompt

Show the answer and why

Answer: System prompt, then examples, then retrieved documents, then the question

The key-value cache is built left to right, so a cached prefix stays valid only up to the first character that changed. Putting the invariant parts first maximises the reusable span; a timestamp at the top invalidates everything.

Question 4. A model card recommends temperature 0.6 with top-p 0.95 and warns against greedy decoding. Your engine defaults to temperature 0.8. What is the risk of leaving the default?

• None: sampler settings only affect speed
• The model will refuse to load
• You are evaluating a configuration the publisher did not test, so a poor result tells you about your settings rather than about the model
• Higher temperature always improves reasoning quality

Show the answer and why

Answer: You are evaluating a configuration the publisher did not test, so a poor result tells you about your settings rather than about the model

Sampling settings are part of the configuration under test. Comparing two models under different, undocumented samplers measures the samplers as much as the models, which is why the lab in this part records temperature, top-p, top-k and seed alongside every score.

Question 5. Which of these are genuine costs of adding eight few-shot examples to every request? Select all that apply.

• Prefill work on every request, unless the prefix is cached and unchanged
• Context length consumed that is then unavailable for the conversation
• A larger model file on disk
• A longer invalid-prefix penalty if anything before the examples changes between requests

Show the answer and why

Answer: Prefill work on every request, unless the prefix is cached and unchanged, Context length consumed that is then unavailable for the conversation, A longer invalid-prefix penalty if anything before the examples changes between requests

Examples live in the prompt, so they cost prefill and context, and they lengthen the prefix that a change at the front invalidates. Nothing about them touches the weights on disk, which is exactly the difference between few-shot prompting and the fine-tuning in Part 13.

Check my answersClear