Lab: Look Inside a Model

Validated on: written from the documentation cited above; not yet validated on hardware on any track. Per-track versions and dates go here once the validation pass has run these scripts.

Objective

Before executing, read the lab execution and evidence guide. Use this lesson's explicit working directories and track setup; keep each server in its own terminal. Record hardware validation as pass, fail or not run, with the evidence requested below.

By the end of this lab you will have a real model on your own disk and will have checked, with four short scripts, every claim the five lessons in this part made about it: every tensor name and shape read from the checkpoint’s headers without loading a weight; the parameter count reconciled with config.json to the digit; the token counts the tokens lesson asked you to predict; the probability the model assigns to each of its 151,936 output entries for one position, under three temperatures; one head’s attention matrix with the causal mask checked as a triangle of exact zeroes; and one question sent to a base checkpoint and to its instruction-tuned sibling.

The model is Qwen3-1.7B, published by Alibaba under the Apache-2.0 licence according to its model card, with Qwen3-1.7B-Base, the same architecture at the end of pretraining, on the same licence; neither is gated. Both are in the model reference, which lists the BF16 weights at about 3.4 GB.

⌂Single-machine path

Single machine on every track, and no accelerator needed: on a CPU the scripts run more slowly and produce the same counts, shapes and zeroes.

What is on the disk when you are done

A model repository is a directory of files, and this lab reads every one of them. The table is the two file lists as the Hub showed them on 2026-09-12, sizes in bytes, with the task that reads each file.

File	Qwen/Qwen3-1.7B	Qwen/Qwen3-1.7B-Base	What it is, and who reads it
config.json	726	727	The architecture as numbers: layers, widths, heads, vocabulary. Task 3, and every script.
model-00001-of-00002.safetensors	3,441,185,608	—	310 tensors: embedding, 28 blocks, final norm. Task 4.
model-00002-of-00002.safetensors	622,329,984	—	One tensor, lm_head.weight. Task 4 explains why it is here.
model.safetensors	—	3,441,185,608	The base checkpoint, all 310 tensors in one file. Task 4.
model.safetensors.index.json	25,605	—	Which tensor lives in which shard, plus total_size. Task 4.
tokenizer.json	11,422,654	7,031,645	The tokeniser: vocabulary, merges, pipeline. Task 5.
vocab.json, merges.txt	2,776,833 and 1,671,853	same	The same vocabulary and merges in the older two-file form.
tokenizer_config.json	9,732	9,678	Special-token names and the chat template. Tasks 5 and 7.
generation_config.json	239	138	Default sampling settings and the end-of-sequence ids. Task 7.
README.md, LICENSE, .gitattributes	13,963, 11,343, 1,570	2,941, 11,343, 1,519	The model card, the Apache-2.0 text, Git LFS settings.

Notice two things before downloading a byte: the instruct repository’s two safetensors files total 4,063,515,592 bytes against the base repository’s 3,441,185,608, 622 MB more for the same architecture; and the base tokenizer.json is 4 MB smaller, although both produce identical ids for every string on this page. Task 4 makes you find the first explanation; task 5 gives the second.

Requirements

Every track needs the course directory and virtual environment from Part 1’s lab, an internet connection, and the disk and time in the table.

Track	Where the scripts run	Memory floor	Download	Free disk needed	Attended	Unattended
S (DGX Spark)	Inside the NGC PyTorch container from Part 1; CUDA	8 GB of the 128 GB	7.53 GB (two repositories)	8 GB	60 to 75 min	the download
X (Ryzen AI Max+)	The Part 1 .venv; ROCm if it worked, otherwise the 16 Zen 5 cores	8 GB	7.53 GB	8 GB	60 to 75 min; add a few minutes per generation on the CPU	the download
M (Apple silicon)	The Part 1 .venv; PyTorch on MPS, and MLX for task 8	8 GB	7.53 GB	8 GB	70 to 85 min with task 8	the download
N (NVIDIA desktop or laptop)	The Part 1 .venv, inside WSL2 on Windows; CUDA, or the CPU on a small card	8 GB	7.53 GB	8 GB	60 to 75 min	the download

The attended figures are the author’s estimate, not measurements, and the download’s wall-clock is arithmetic on your link:

Your link, as the provider states it	7.53 GB takes about
50 Mb/s	20 minutes
100 Mb/s	10 minutes
500 Mb/s	2 minutes
1 Gb/s	1 minute

The scripts load the weights in BF16, their native precision, so about 3.4 GB is resident while one runs; task 7 loads the two models one after the other, never together.

Below 8 GB, or short of disk. The same four scripts run unchanged on Qwen3-0.6B (Qwen/Qwen3-0.6B and Qwen/Qwen3-0.6B-Base, Apache-2.0, not gated, about 1.5 GB and 1.2 GB on disk):

RunnableAll tracks

reduced path: download the 0.6B pair instead

hf download Qwen/Qwen3-0.6B --local-dir ~/llm-course/models/qwen3-0.6b
hf download Qwen/Qwen3-0.6B-Base --local-dir ~/llm-course/models/qwen3-0.6b-base

Every --model, --base and --instruct path on this page then ends in qwen3-0.6b or qwen3-0.6b-base; the counts change, the shapes shrink from 2048 to 1024 wide, and every mechanism is the same. The dry runs behind this page’s sample outputs used those two, because the author’s writing machine had no accelerator and a 2 GB download limit, and each such block says so.

The versions this page was written against are transformers 5.16.1 · verified 2026-09-08, Hugging Face CLI 1.30.0 · verified 2026-09-08, uv 0.12.11 · verified 2026-09-08 and, for task 8, mlx-lm 0.31.3 · verified 2026-09-08.

Track S — NVIDIA DGX Spark

Work inside the NGC PyTorch container that Part 1’s setup-env-spark.sh starts, with your course directory mounted at /workspace/course. The container has no uv environment, so install the libraries with its own pip, once per container start unless you commit the image:

RunnableTrack S · DGX Spark

inside the container: add this part's libraries

pip install transformers tokenizers safetensors accelerate huggingface_hub

Two substitutions apply to every command on this page: where the page writes ~/llm-course, type /workspace/course, and skip every source .venv/bin/activate line; after the pip install above, python and hf are the container’s own. The downloads land on the host and survive the container.

Track X — AMD Ryzen AI Max+ 395Partial

The ROCm path for this chip uses AMD's nightly wheel index, as Part 1 recorded. The lab is small enough that the CPU fallback finishes comfortably, and every count, shape and zero is identical either way.

Activate the Part 1 environment and install the libraries into it:

RunnableTrack X · Ryzen AI Max+

add this part's libraries to the Part 1 environment

cd ~/llm-course
source .venv/bin/activate
uv pip install transformers tokenizers safetensors accelerate huggingface_hub

With a working ROCm build from Part 1 the scripts print device: cuda, PyTorch’s name for an AMD GPU through the ROCm build. With device: cpu, finish the lab on the CPU: only task 7’s eighty-token generation makes you wait, a minute or two per model.

Track M — Apple silicon

Activate the Part 1 environment and install the libraries into it, plus MLX LM for task 8:

RunnableTrack M · Apple silicon

add this part's libraries and MLX LM to the Part 1 environment

cd ~/llm-course
source .venv/bin/activate
uv pip install transformers tokenizers safetensors accelerate huggingface_hub mlx-lm

The scripts select PyTorch’s MPS backend and print device: mps; task 8 runs the same generation through MLX against the same directory, so there is no second download. Stay with Transformers for task 6: in mlx-lm 0.31.3 the Qwen3 attention layer calls mx.fast.scaled_dot_product_attention and returns only its output (mlx_lm/models/qwen3.py and models/base.py at the v0.31.3 tag, read 2026-09-12), so there is no attention matrix for it to hand back.

Track N — NVIDIA desktop or laptop

Activate the Part 1 environment, inside WSL2 on Windows, and install the libraries into it:

RunnableTrack N · NVIDIA GPU

add this part's libraries to the Part 1 environment

cd ~/llm-course
source .venv/bin/activate
uv pip install transformers tokenizers safetensors accelerate huggingface_hub

The scripts print device: cuda when PyTorch sees the GPU. The weights are 3.44 GB in BF16 with the CUDA context on top, so a card with 6 GB or more is the comfortable case, an estimate rather than a measurement; on a smaller card, or after CUDA out of memory, prefix the command with CUDA_VISIBLE_DEVICES="" to hide the GPU and finish on the CPU. On Windows the only driver is the Windows one, as Part 1’s lab set out.

Before you start

Where the files live, first. Download the four lab files embedded in tasks 4 to 7 into ~/llm-course (Track S: /workspace/course) and save the two scripts printed in tasks 3 and 4 there under the names in their titles. Every python command on this page runs from that directory, which is why --labbook labbook.md is a relative path to the notebook Part 1 created there. Then five checks, each with the line that means you may go on.

RunnableAll tracks

preflight 1: the environment and its versions

cd ~/llm-course
source .venv/bin/activate
python -c "import torch, transformers, safetensors, tokenizers, huggingface_hub; print('torch', torch.__version__); print('transformers', transformers.__version__); print('safetensors', safetensors.__version__); print('tokenizers', tokenizers.__version__); print('huggingface_hub', huggingface_hub.__version__)"

Output — what you should see

torch 2.x.x+cu13x        (or +rocmx.x on Track X, or 2.x.x with no suffix on a Mac)
transformers 5.16.1      (5.16 or later; the model card requires at least 4.51.0)
safetensors 0.x.x
tokenizers 0.x.x
huggingface_hub 1.30.0   (1.30 or later)

Record the five version lines in the notebook’s Environment section. If the import fails, the install step in your track’s tab did not run in this environment; see Troubleshooting.

RunnableAll tracks

preflight 2: the CLI, where it is, and whether you are signed in

command -v hf
hf version
hf auth whoami

Output — what you should see

/home/you/llm-course/.venv/bin/hf
✓ hf version
  version: 1.30.0
Error: Not logged in

command -v hf must print a path inside the environment you installed into: the .venv on Tracks X, M and N, or on Track S a path inside the container. The documentation says hf auth whoami “simply prints your username and the organizations you are a part of” and that “if you are not logged in, an error message will be printed”; for a public model that changes nothing, and task 1 says when it is worth fixing. The first hf command of a day may print a grey Hint: line or two before its output (a newer release exists; an agent skill is not installed); neither is an error. If you see version=1.30.0 instead, and later a bare path=… line or tab-separated tables, that is the CLI’s agent format, printed when it detects an AI coding agent’s environment variables; a terminal shows the ✓ hf version / version: 1.30.0 and ✓ Downloaded / path: … pairs, as the blocks on this page do.

RunnableAll tracks

preflight 3: the accelerator PyTorch will use

python -c "import torch; print('cuda', torch.cuda.is_available()); mps = getattr(torch.backends, 'mps', None); print('mps', mps is not None and mps.is_available())"

Output — what you should see

cuda True
mps False

One of the two is True on Tracks S, M and N and on Track X when ROCm works; both False means the CPU, which finishes the lab. Record which.

RunnableAll tracks

preflight 4: disk

df -h ~/llm-course

The Avail column must show at least 8G; if not, read the reduced path in Requirements before downloading anything.

RunnableAll tracks

preflight 5: what the download will be, without downloading it

mkdir -p ~/llm-course/models
hf download Qwen/Qwen3-1.7B --dry-run --local-dir ~/llm-course/models/qwen3-1.7b
hf download Qwen/Qwen3-1.7B-Base --dry-run --local-dir ~/llm-course/models/qwen3-1.7b-base

The documentation describes --dry-run as listing “all files to download on the repo” and checking “whether they are already downloaded or not”; nothing is written except the empty directories. The two Will download lines are the ones to read:

Output — what you should see

[dry-run] Fetching 12 files: 100%|██████████| 12/12 [00:00<00:00, xx.xxit/s]
[dry-run] Will download 12 files (out of 12) totalling 4.1G.
FILE                             SIZE
-------------------------------- ------
.gitattributes                   1.6K
LICENSE                          11.3K
README.md                        14.0K
config.json                      726.0
generation_config.json           239.0
merges.txt                       1.7M
model-00001-of-00002.safetensors 3.4G
model-00002-of-00002.safetensors 622.3M
model.safetensors.index.json     25.6K
tokenizer.json                   11.4M
tokenizer_config.json            9.7K
vocab.json                       2.8M
[dry-run] Fetching 10 files: 100%|██████████| 10/10 [00:00<00:00, xx.xxit/s]
[dry-run] Will download 10 files (out of 10) totalling 3.5G.
FILE                   SIZE
---------------------- -----
...
model.safetensors      3.4G
tokenizer.json         7.0M
...

Task 2’s unauthenticated-requests warning may appear between the lines. If the counts are not twelve and ten, the repositories have changed since 2026-09-12: note the new file list in the notebook and read the tasks with the new names.

Tasks

1. Sign in to the Hub, if you need to

Qwen3-1.7B is public and downloads without an account, but signing in once is worth it: several later models are gated, and the CLI warns about rate limits on anonymous downloads. The documentation describes hf auth login as logging you in with your browser: “it prints a URL and a short code. Open the URL, enter the code, approve the request, and the CLI retrieves and saves an access token on your machine.”

RunnableAll tracks

optional: authenticate the CLI

hf auth login

Output — what you should see

? How would you like to log in?  [Use arrows, Enter to confirm]
> Log in with your browser
  Paste an access token

    Open this URL in your browser:
        https://huggingface.co/oauth/device

    And enter the code: XXXX-XXXX

    Waiting for authorization...
Token is valid.
Your token has been saved to /home/you/.cache/huggingface/token
Login successful.

Afterwards hf auth whoami prints your username. Record only whether you signed in; the token is a credential and stays out of the notebook.

⛨Security

The documentation also accepts a token on the command line and recommends an environment variable instead, “to avoid pasting it in your command history”. Prefer the browser flow; if you create a token, give it read-only scope, and know that hf auth login --add-to-git-credential also stores it in your Git credential helper.

2. Download the two checkpoints

hf download fetches a repository and, as the documentation puts it, “prints the returned path to the terminal”. By default files go into the shared cache under HF_HOME; --local-dir puts them in a directory you name, so that every model in the notebook has an obvious path.

RunnableAll tracks

download the instruction-tuned checkpoint

hf download Qwen/Qwen3-1.7B --local-dir ~/llm-course/models/qwen3-1.7b

Output — what you should see

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Fetching 12 files: 100%|██████████████████████████| 12/12 [xx:xx<00:00, x.xxs/it]
✓ Downloaded
  path: /home/you/llm-course/models/qwen3-1.7b

The warning appears only if you skipped task 1 and is not an error; the progress bars, of which 1.30.0 may draw more than one, are the CLI’s own and may differ by version. The last two lines are the path the documentation says is always printed. If the download stops partway, run the same command again: the CLI keeps metadata under .cache/huggingface/ inside the directory, which the documentation says “prevents re-downloading files if they’re already up-to-date”.

Now the base checkpoint, which task 7 needs:

RunnableAll tracks

download the base checkpoint

hf download Qwen/Qwen3-1.7B-Base --local-dir ~/llm-course/models/qwen3-1.7b-base

Look at what arrived, hidden entries included:

RunnableAll tracks

see what a model repository actually contains

ls -lA ~/llm-course/models/qwen3-1.7b ~/llm-course/models/qwen3-1.7b-base

Output — what you should see

/home/you/llm-course/models/qwen3-1.7b:
total xxxxxxx
drwxr-xr-x 3 you you       xxxx ... .cache
-rw-r--r-- 1 you you       1570 ... .gitattributes
...
-rw-r--r-- 1 you you 3441185608 ... model-00001-of-00002.safetensors
-rw-r--r-- 1 you you  622329984 ... model-00002-of-00002.safetensors
...

/home/you/llm-course/models/qwen3-1.7b-base:
total xxxxxxx
...
-rw-r--r-- 1 you you 3441185608 ... model.safetensors
...

Twelve and ten entries counting .gitattributes, plus the .cache directory the CLI adds, and the byte counts of the table at the top, because they are the same bytes. That directory is the whole model: weights, architecture, tokeniser and chat template, as files.

✦Tip

Short of disk for the second repository? hf download Qwen/Qwen3-1.7B-Base --include "*.json" --local-dir ~/llm-course/models/qwen3-1.7b-base fetches the configuration and tokeniser without the weights, enough for tasks 3 and 5; task 7 then runs the instruct model against itself, as Troubleshooting describes.

Record in the notebook: the two directory paths, the date, the twelve and ten entry names, and the two safetensors byte counts.

3. Read the configuration, and predict the count

Open the file the last lesson described:

RunnableAll tracks

read config.json

cat ~/llm-course/models/qwen3-1.7b/config.json

The file as retrieved on 2026-09-12, cut to the fields this task uses; the real one adds rope_theta, rms_norm_eps, dropout and window settings:

Qwen/Qwen3-1.7B — config.json (extract, retrieved 2026-09-12)

{
  "architectures": ["Qwen3ForCausalLM"],
  "attention_bias": false,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "head_dim": 128,
  "hidden_size": 2048,
  "intermediate_size": 6144,
  "max_position_embeddings": 40960,
  "num_attention_heads": 16,
  "num_hidden_layers": 28,
  "num_key_value_heads": 8,
  "tie_word_embeddings": true,
  "torch_dtype": "bfloat16",
  "vocab_size": 151936
}

Do the arithmetic before the next task checks it. The parameters lesson did two sums from config.json; here are all of them for one block. A linear layer’s weight is stored as (output width, input width), and these are the shapes task 4 will print:

Tensor in one block	Shape, from the config fields	Parameters
self_attn.q_proj.weight	(16 × 128, 2048) = (2048, 2048)	4,194,304
self_attn.k_proj.weight	(8 × 128, 2048) = (1024, 2048)	2,097,152
self_attn.v_proj.weight	(1024, 2048)	2,097,152
self_attn.o_proj.weight	(2048, 16 × 128) = (2048, 2048)	4,194,304
self_attn.q_norm.weight, self_attn.k_norm.weight	(128,) each	256
mlp.gate_proj.weight, mlp.up_proj.weight	(6144, 2048) each	25,165,824
mlp.down_proj.weight	(2048, 6144)	12,582,912
input_layernorm.weight, post_attention_layernorm.weight	(2048,) each	4,096
One block	11 tensors	50,336,000

Attention is 12,583,168 of the 50,336,000 and the feed-forward network 37,748,736, three quarters of the block, as the parameters lesson promised. Then the stack:

non-embedding = 28 blocks × 50,336,000 + model.norm (2,048)           = 1,409,410,048
embedding     = vocab_size × hidden_size = 151,936 × 2,048             =   311,164,928
total         = non-embedding + embedding                              = 1,720,574,976

The card’s 1.4B and 1.7B are these numbers before rounding. Save the same arithmetic as ~/llm-course/count-from-config.py, to point at any Qwen3 config.json in the course before downloading its weights:

RunnableAll tracks

count-from-config.py

import json, sys
c = json.load(open(sys.argv[1]))                      # path to config.json
h, inter = c["hidden_size"], c["intermediate_size"]
q_width = c["num_attention_heads"] * c["head_dim"]    # 16 x 128 = 2048
kv_width = c["num_key_value_heads"] * c["head_dim"]   # 8 x 128 = 1024
attention = 2 * h * q_width + 2 * h * kv_width + 2 * c["head_dim"]   # q,o + k,v + q_norm,k_norm
mlp = 3 * h * inter                                   # gate, up, down
norms = 2 * h                                         # input_layernorm, post_attention_layernorm
per_layer = attention + mlp + norms
non_embedding = per_layer * c["num_hidden_layers"] + h   # + model.norm
embedding = c["vocab_size"] * h
print(f"attention per layer   {attention:>15,}")
print(f"feed-forward per layer{mlp:>15,}")
print(f"per layer             {per_layer:>15,}")
print(f"non-embedding         {non_embedding:>15,}   (model card: 1.4B)")
print(f"embedding matrix      {embedding:>15,}")
print(f"total, head tied      {non_embedding + embedding:>15,}   (model card: 1.7B)")
print(f"BF16 bytes once loaded{2 * (non_embedding + embedding):>15,}")

RunnableAll tracks

run it on the configuration you just read

python count-from-config.py ~/llm-course/models/qwen3-1.7b/config.json

Output — what you should see

attention per layer        12,583,168
feed-forward per layer     37,748,736
per layer                  50,336,000
non-embedding           1,409,410,048   (model card: 1.4B)
embedding matrix          311,164,928
total, head tied        1,720,574,976   (model card: 1.7B)
BF16 bytes once loaded  3,441,149,952

The last line is Part 1’s bytes-per-parameter arithmetic: two bytes per BF16 parameter. Now write down a prediction and keep it: how many bytes of tensor data do you expect the instruct repository’s two safetensors files to hold? You know their sizes from the table at the top; if the two disagree, you have found what task 4 is about.

Record in the notebook: the seven config fields, the per-block count, the three totals, and your byte prediction.

4. List the tensors and reconcile the count

The script reads only the safetensors headers: as the format specification describes, a file begins with eight bytes giving the header length as an unsigned little-endian 64-bit integer, then that many bytes of JSON naming every tensor with its dtype, shape and data_offsets, then the byte buffer. Listing a multi-gigabyte checkpoint therefore reads a few tens of kilobytes, with nothing but the standard library.

RunnableAll tracks

list-tensors.py

"""List every tensor in a safetensors checkpoint and count the parameters.

Purpose: open the safetensors file(s) of a downloaded model, print each tensor's name,
         shape and dtype, total the parameters and bytes, and compare the file with the
         arithmetic that config.json predicts, so that the numbers on a model card can be
         checked against the checkpoint on your own disk.
Platform: all (this reads file headers only; no accelerator and no model loading)
Minimum memory: 8 GB (the script itself needs a few megabytes)
Assumes: a model directory downloaded with `hf download ... --local-dir <dir>` containing
         one or more *.safetensors files and a config.json; the course environment from
         Part 1. Only the standard library is used.

Usage: python list-tensors.py --model ~/llm-course/models/qwen3-1.7b [--labbook labbook.md]
           [--limit 12] [--layer 0]
"""
import argparse
import json
import struct
from pathlib import Path

# Bytes per element for the dtype strings the safetensors header uses.
DTYPE_BYTES = {
    "BOOL": 1, "U8": 1, "I8": 1, "F8_E4M3": 1, "F8_E5M2": 1,
    "I16": 2, "U16": 2, "F16": 2, "BF16": 2,
    "I32": 4, "U32": 4, "F32": 4,
    "I64": 8, "U64": 8, "F64": 8,
}


def read_header(path: Path) -> tuple[dict, int]:
    """Read a safetensors header without touching the weights.

    The format specification says the file starts with 8 bytes holding an unsigned
    little-endian 64-bit integer N, followed by N bytes of JSON, followed by the byte
    buffer. Reading the first 8 + N bytes is therefore enough to list everything.
    Returns the parsed header and N.
    """
    with path.open("rb") as fh:
        (header_len,) = struct.unpack("<Q", fh.read(8))
        header = json.loads(fh.read(header_len).decode("utf-8"))
    return header, header_len


def numel(shape: list[int]) -> int:
    n = 1
    for dim in shape:
        n *= dim
    return n


def expected_from_config(config: dict) -> dict | None:
    """Parameter arithmetic for a dense Qwen3-style block, from config.json alone.

    Per block: q_proj (heads*head_dim x hidden), k_proj and v_proj (kv_heads*head_dim x
    hidden), o_proj (hidden x heads*head_dim), q_norm and k_norm (head_dim each),
    gate_proj and up_proj (intermediate x hidden), down_proj (hidden x intermediate),
    two RMSNorm scales (hidden each). Then the final norm and the embedding matrix.
    Returns None when the config lacks a field or declares attention biases.
    """
    fields = ("hidden_size", "intermediate_size", "num_hidden_layers", "num_attention_heads",
              "num_key_value_heads", "head_dim", "vocab_size")
    if any(f not in config for f in fields) or config.get("attention_bias"):
        return None
    h, inter, layers = config["hidden_size"], config["intermediate_size"], config["num_hidden_layers"]
    q_width = config["num_attention_heads"] * config["head_dim"]
    kv_width = config["num_key_value_heads"] * config["head_dim"]
    attention = 2 * h * q_width + 2 * h * kv_width + 2 * config["head_dim"]
    mlp = 3 * h * inter
    norms = 2 * h
    per_layer = attention + mlp + norms
    embedding = config["vocab_size"] * h
    non_embedding = per_layer * layers + h
    return {
        "attention_per_layer": attention, "mlp_per_layer": mlp, "norms_per_layer": norms,
        "per_layer": per_layer, "non_embedding": non_embedding, "embedding": embedding,
        "total_tied": non_embedding + embedding, "total_untied": non_embedding + 2 * embedding,
    }


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--model", required=True, help="directory holding the checkpoint")
    parser.add_argument("--limit", type=int, default=12, help="tensors to print in full before summarising")
    parser.add_argument("--layer", type=int, default=None, help="also print every tensor of this layer index")
    parser.add_argument("--labbook", default=None, help="append one JSON line per run to this file")
    args = parser.parse_args()

    model_dir = Path(args.model).expanduser()
    if not model_dir.is_dir():
        raise SystemExit(f"{model_dir} is not a directory; check the --model path")
    files = sorted(model_dir.glob("*.safetensors"))
    if not files:
        raise SystemExit(f"no *.safetensors files in {model_dir}; the download did not finish, or the path is wrong")

    config_path = model_dir / "config.json"
    config = json.loads(config_path.read_text(encoding="utf-8")) if config_path.exists() else {}
    if not config:
        print(f"warning: no config.json in {model_dir}; the config comparison is skipped")

    tensors: list[tuple[str, list[int], str, int, str]] = []
    metadata: dict[str, str] = {}
    per_file: list[dict] = []
    for path in files:
        header, header_len = read_header(path)
        meta = header.pop("__metadata__", None)
        if isinstance(meta, dict):
            metadata.update({str(k): str(v) for k, v in meta.items()})
        count = 0
        for name, info in header.items():
            shape = [int(d) for d in info["shape"]]
            tensors.append((name, shape, info["dtype"], numel(shape), path.name))
            count += 1
        per_file.append({"file": path.name, "bytes": path.stat().st_size, "header_bytes": header_len, "tensors": count})

    tensors.sort(key=lambda t: t[0])
    total_params = sum(t[3] for t in tensors)
    total_bytes = sum(t[3] * DTYPE_BYTES.get(t[2], 0) for t in tensors)
    dtypes = sorted({t[2] for t in tensors})
    names = {t[0] for t in tensors}

    # The embedding matrix is the tensor whose first dimension is the vocabulary; an
    # output head, if stored, has the same shape under a different name.
    vocab_size = config.get("vocab_size")
    vocab_sized = [t for t in tensors if vocab_size and t[1] and t[1][0] == vocab_size]
    embedding_params = sum(t[3] for t in vocab_sized)
    tied = bool(config.get("tie_word_embeddings"))
    lm_head_stored = "lm_head.weight" in names
    lm_head_params = next((t[3] for t in tensors if t[0] == "lm_head.weight"), 0)
    # What the model occupies once loaded: a tied head is one matrix in memory even
    # when the file stores it twice.
    unique_params = total_params - (lm_head_params if tied and lm_head_stored else 0)
    bytes_per_param = DTYPE_BYTES.get(dtypes[0], 0) if len(dtypes) == 1 else None

    print(f"directory:  {model_dir}")
    for f in per_file:
        print(f"file:       {f['file']:36s} {f['bytes']:>15,} bytes on disk   "
              f"header {f['header_bytes']:>7,} bytes   {f['tensors']} tensors")
    print(f"tensors:    {len(tensors)}")
    print(f"dtypes:     {', '.join(dtypes)}")
    if metadata:
        print(f"metadata:   {metadata}")
    print()
    for name, shape, dtype, n, _ in tensors[: args.limit]:
        print(f"  {name:52s} {str(tuple(shape)):>22s}  {dtype:5s}  {n:>13,}")
    if len(tensors) > args.limit:
        print(f"  ... and {len(tensors) - args.limit} more tensors")

    layer_params = None
    if args.layer is not None:
        prefix = f"model.layers.{args.layer}."
        block = [t for t in tensors if t[0].startswith(prefix)]
        if not block:
            raise SystemExit(f"no tensors named {prefix}*; the model has {config.get('num_hidden_layers', '?')} layers")
        layer_params = sum(t[3] for t in block)
        print(f"\nevery tensor of layer {args.layer} ({len(block)} tensors, {layer_params:,} parameters):")
        for name, shape, dtype, n, _ in block:
            print(f"  {name[len(prefix):]:40s} {str(tuple(shape)):>16s}  {dtype:5s}  {n:>13,}")

    print()
    print(f"parameters in the file:        {total_params:>15,}")
    print(f"  in vocabulary-sized tensors: {embedding_params:>15,}   "
          f"({len(vocab_sized)} tensor(s): {', '.join(t[0] for t in vocab_sized) or 'none'})")
    print(f"  everything else:             {total_params - embedding_params:>15,}")
    print(f"tie_word_embeddings: {tied}   lm_head.weight stored in the file: {lm_head_stored}")
    if tied and lm_head_stored:
        print(f"  the output head is a second copy of the embedding matrix; it is loaded once, so")
        print(f"  parameters once loaded:      {unique_params:>15,}")
    print(f"weight bytes on disk:          {total_bytes:>15,}  ({total_bytes / 1e9:.2f} GB)")
    if bytes_per_param:
        print(f"weight bytes once loaded:      {unique_params * bytes_per_param:>15,}  "
              f"({unique_params * bytes_per_param / 1e9:.2f} GB at {bytes_per_param} bytes per parameter)")

    expected = expected_from_config(config) if config else None
    if config:
        print()
        for field in ("model_type", "num_hidden_layers", "hidden_size", "intermediate_size",
                      "num_attention_heads", "num_key_value_heads", "head_dim",
                      "vocab_size", "max_position_embeddings", "tie_word_embeddings", "torch_dtype"):
            if field in config:
                print(f"  config {field:26s} {config[field]}")
    if expected:
        print()
        print("from config.json alone (dense Qwen3-style block):")
        print(f"  attention per layer:  {expected['attention_per_layer']:>15,}")
        print(f"  feed-forward per layer:{expected['mlp_per_layer']:>14,}")
        print(f"  norms per layer:      {expected['norms_per_layer']:>15,}")
        print(f"  per layer:            {expected['per_layer']:>15,}"
              + (f"   file says {layer_params:,}  match: {layer_params == expected['per_layer']}" if layer_params else ""))
        print(f"  non-embedding:        {expected['non_embedding']:>15,}   file says {total_params - embedding_params:,}"
              f"  match: {expected['non_embedding'] == total_params - embedding_params}")
        print(f"  embedding matrix:     {expected['embedding']:>15,}")
        print(f"  total, tied head:     {expected['total_tied']:>15,}")
        print(f"  total, untied head:   {expected['total_untied']:>15,}")

    if args.labbook:
        record = {
            "lab": "part-02/list-tensors",
            "model_dir": str(model_dir),
            "files": per_file,
            "tensors": len(tensors),
            "dtypes": dtypes,
            "parameters_in_file": total_params,
            "parameters_loaded": unique_params,
            "embedding_parameters": embedding_params,
            "lm_head_stored": lm_head_stored,
            "weight_bytes_on_disk": total_bytes,
            "config": {k: config.get(k) for k in ("num_hidden_layers", "hidden_size", "intermediate_size",
                                                  "num_attention_heads", "num_key_value_heads",
                                                  "head_dim", "vocab_size", "tie_word_embeddings")},
            "expected_from_config": expected,
        }
        with Path(args.labbook).expanduser().open("a", encoding="utf-8") as fh:
            fh.write(json.dumps(record) + "\n")
        print(f"\nrecorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download list-tensors.py218 lines

RunnableAll tracks

list the tensors of the instruct checkpoint, with every tensor of block 0

python list-tensors.py --model ~/llm-course/models/qwen3-1.7b --layer 0 --labbook labbook.md

Every figure below is derived from the repository’s index file, the Hub’s file sizes and config.json as retrieved on 2026-09-12, so it should match to the digit; the file name column and the date are yours.

Output — what you should see

directory:  /home/you/llm-course/models/qwen3-1.7b
file:       model-00001-of-00002.safetensors       3,441,185,608 bytes on disk   header  35,648 bytes   310 tensors
file:       model-00002-of-00002.safetensors         622,329,984 bytes on disk   header     120 bytes   1 tensors
tensors:    311
dtypes:     BF16
metadata:   {'format': 'pt'}

  lm_head.weight                                               (151936, 2048)  BF16     311,164,928
  model.embed_tokens.weight                                    (151936, 2048)  BF16     311,164,928
  model.layers.0.input_layernorm.weight                               (2048,)  BF16           2,048
  model.layers.0.mlp.down_proj.weight                            (2048, 6144)  BF16      12,582,912
  model.layers.0.mlp.gate_proj.weight                            (6144, 2048)  BF16      12,582,912
  model.layers.0.mlp.up_proj.weight                              (6144, 2048)  BF16      12,582,912
  model.layers.0.post_attention_layernorm.weight                      (2048,)  BF16           2,048
  model.layers.0.self_attn.k_norm.weight                               (128,)  BF16             128
  model.layers.0.self_attn.k_proj.weight                         (1024, 2048)  BF16       2,097,152
  model.layers.0.self_attn.o_proj.weight                         (2048, 2048)  BF16       4,194,304
  model.layers.0.self_attn.q_norm.weight                               (128,)  BF16             128
  model.layers.0.self_attn.q_proj.weight                         (2048, 2048)  BF16       4,194,304
  ... and 299 more tensors

every tensor of layer 0 (11 tensors, 50,336,000 parameters):
  input_layernorm.weight                            (2048,)  BF16           2,048
  ...
  self_attn.v_proj.weight                      (1024, 2048)  BF16       2,097,152

parameters in the file:          2,031,739,904
  in vocabulary-sized tensors:     622,329,856   (2 tensor(s): lm_head.weight, model.embed_tokens.weight)
  everything else:               1,409,410,048
tie_word_embeddings: True   lm_head.weight stored in the file: True
  the output head is a second copy of the embedding matrix; it is loaded once, so
  parameters once loaded:        1,720,574,976
weight bytes on disk:            4,063,479,808  (4.06 GB)
weight bytes once loaded:        3,441,149,952  (3.44 GB at 2 bytes per parameter)

  config model_type                 qwen3
  config num_hidden_layers          28
  ...
  config tie_word_embeddings        True
  config torch_dtype                bfloat16

from config.json alone (dense Qwen3-style block):
  attention per layer:       12,583,168
  feed-forward per layer:    37,748,736
  norms per layer:                4,096
  per layer:                 50,336,000   file says 50,336,000  match: True
  non-embedding:          1,409,410,048   file says 1,409,410,048  match: True
  embedding matrix:         311,164,928
  total, tied head:       1,720,574,976
  total, untied head:     2,031,739,904

recorded in labbook.md

Read it against task 3. Block 0 is your table: eleven tensors, the shapes you predicted, 50,336,000 parameters, match: True on the per-layer and non-embedding lines. The file agrees with the configuration to the digit.

The file holds 2,031,739,904 parameters, not 1,720,574,976. The difference is exactly 311,164,928, one embedding matrix, and the listing shows where: lm_head.weight, (151936, 2048), the entire second shard. config.json says tie_word_embeddings: true, so the loader uses one matrix for both the input embedding and the output projection and “parameters once loaded” is the card’s 1.7B; the publisher’s export wrote the tied copy to disk anyway. That is why the instruct repository is 622 MB larger than the base one, whose single file you will find has no lm_head.weight, and why the model reference figure of 3.4 GB describes memory rather than the download. Reading a repository’s size as a parameter count, or the reverse, gets this model wrong by 18 percent: memory follows “parameters once loaded” times bytes per parameter, disk follows the file list, and Part 4’s lab keeps the two apart.

The header is 35,648 bytes, which you can derive from the file size, 3,441,185,608 − 8 − 3,441,149,952 bytes of tensor data. Everything the script printed came from those bytes and the second shard’s 120.

Now the base checkpoint:

RunnableAll tracks

list the base checkpoint

python list-tensors.py --model ~/llm-course/models/qwen3-1.7b-base --limit 3 --labbook labbook.md

Output — what you should see

directory:  /home/you/llm-course/models/qwen3-1.7b-base
file:       model.safetensors                      3,441,185,608 bytes on disk   header  35,648 bytes   310 tensors
tensors:    310
dtypes:     BF16
...
parameters in the file:          1,720,574,976
  in vocabulary-sized tensors:     311,164,928   (1 tensor(s): model.embed_tokens.weight)
  everything else:               1,409,410,048
tie_word_embeddings: True   lm_head.weight stored in the file: False
weight bytes on disk:            3,441,149,952  (3.44 GB)
weight bytes once loaded:        3,441,149,952  (3.44 GB at 2 bytes per parameter)
...

Same architecture, same 1,409,410,048 non-embedding parameters, 310 tensors instead of 311, a file byte-for-byte the size of the instruct repository’s first shard. The two checkpoints differ in the values of their weights, which is what post-training changed, and in one exported copy of a matrix, which is packaging.

Go one level further, optionally. The safetensors documentation shows safe_open with get_slice, which reads a rectangle of one tensor and nothing else. The embeddings lesson said a token id is a row number; save this as ~/llm-course/embedding-row.py to read four such rows straight from the shard and compare them by cosine similarity:

RunnableAll tracks

embedding-row.py

import sys
from pathlib import Path
import torch
from safetensors import safe_open
from transformers import AutoTokenizer

model_dir = Path(sys.argv[1]).expanduser()
tok = AutoTokenizer.from_pretrained(model_dir)
words = ["ĠParis", "ĠLondon", "ĠFrance", "Ġbanana"]
ids = {w: tok.convert_tokens_to_ids(w) for w in words}
for path in sorted(model_dir.glob("*.safetensors")):          # find the shard holding the embedding
    with safe_open(path, framework="pt", device="cpu") as f:
        if "model.embed_tokens.weight" not in f.keys():
            continue
        rows = f.get_slice("model.embed_tokens.weight")        # nothing is read yet
        print("embedding matrix", rows.get_shape(), "in", path.name)
        vec = {w: rows[i:i + 1].float()[0] for w, i in ids.items()}   # one row each, 4 KB of I/O
for w in words[1:]:
    cos = torch.nn.functional.cosine_similarity(vec["ĠParis"], vec[w], dim=0).item()
    print(f"cos(row {ids['ĠParis']} 'ĠParis', row {ids[w]} {w!r}) = {cos:.3f}")

RunnableAll tracks

read four rows of the embedding matrix

python embedding-row.py ~/llm-course/models/qwen3-1.7b

Output — what you should see

embedding matrix [151936, 2048] in model-00001-of-00002.safetensors
cos(row 12095 'ĠParis', row 7148 'ĠLondon') = 0.xxx
cos(row 12095 'ĠParis', row 9625 'ĠFrance') = 0.xxx
cos(row 12095 'ĠParis', row 43096 'Ġbanana') = 0.xxx

The row numbers are exact, because they are the tokeniser’s ids; the cosines are yours to read. On the author’s 0.6B dry run the city and country rows sat well above the fruit row, the static-embedding geometry the embeddings lesson described, and no more than that.

Record in the notebook: the script appended a JSON line per run; add by hand the sentence that explains the 311,164,928 difference, in your own words, because the capstone will ask for it.

5. Tokenise, and be wrong about something

RunnableAll tracks

tokenise-samples.py

"""Tokenise several strings with a model's own tokeniser and show what it did.

Purpose: turn the tokens lesson into measurements: how many tokens the same meaning costs
         in English and Portuguese, what happens to source code and emoji, how the
         byte-level vocabulary displays non-ASCII text, and what the special and chat
         control tokens of this model are.
Platform: all (CPU only; the tokeniser is data, not a model)
Minimum memory: 8 GB (the tokeniser itself needs a few hundred megabytes)
Assumes: a model directory downloaded with `hf download ... --local-dir <dir>` containing
         tokenizer.json and tokenizer_config.json; transformers and tokenizers installed
         in the course environment from Part 1.

Usage: python tokenise-samples.py --model ~/llm-course/models/qwen3-1.7b
           [--guess 4,6,8,2] [--text "your own string"] [--labbook labbook.md]
"""
import argparse
import json
from pathlib import Path

from transformers import AutoTokenizer

# The four strings the tokens lesson asked you to predict, in that order.
PREDICTIONS = [
    ("hello", "Hello, world!"),
    ("hello-pt", "Olá, mundo!"),
    ("signature", "def get_user_by_id(user_id: int):"),
    ("flag-rocket", "🇧🇷🚀"),
]

SAMPLES = [
    ("english", "The quick brown fox jumps over the lazy dog near the river bank."),
    ("portuguese", "A rápida raposa castanha salta sobre o cão preguiçoso perto da margem do rio."),
    ("code", "def get_user_by_id(user_id: int) -> User | None:\n    return session.get(User, user_id)\n"),
    ("emoji", "🇧🇷 🚀 ✨ 🧮"),
]

# Chat control tokens named in the Qwen3 chat template; printed if the vocabulary has them.
CONTROL_TOKENS = ["<|endoftext|>", "<|im_start|>", "<|im_end|>", "<think>", "</think>"]


def describe(tokeniser, label: str, text: str, guess: int | None = None) -> dict:
    """Tokenise one string and print ids, token strings and the three counts."""
    ids = tokeniser(text, add_special_tokens=False)["input_ids"]
    pieces = tokeniser.convert_ids_to_tokens(ids)
    n_bytes = len(text.encode("utf-8"))
    n_words = len([w for w in text.split() if w])
    print(f"\n[{label}]  {text!r}")
    guessed = f"   your guess {guess:4d}" if guess is not None else ""
    print(f"  tokens {len(ids):4d}   words {n_words:4d}   characters {len(text):4d}   utf-8 bytes {n_bytes:4d}{guessed}")
    print(f"  ids    {ids}")
    print(f"  pieces {pieces}")
    if not text.isascii():
        # Byte-level BPE shows each UTF-8 byte as one printable character, so 'á' (bytes
        # C3 A1) appears as 'á'. Decoding each piece on its own shows the text it holds.
        decoded = [tokeniser.decode([i]) for i in ids]
        print(f"  pieces decoded one by one {decoded}")
    if n_words:
        print(f"  tokens per word {len(ids) / n_words:.2f}   bytes per token {n_bytes / max(len(ids), 1):.2f}")
    # Round-tripping proves the split is lossless for this tokeniser.
    restored = tokeniser.decode(ids)
    print(f"  decode round-trip identical: {restored == text}")
    return {
        "label": label, "tokens": len(ids), "words": n_words, "characters": len(text),
        "utf8_bytes": n_bytes, "round_trip": restored == text, "guess": guess,
    }


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--model", required=True, help="model directory or Hub id")
    parser.add_argument("--guess", default=None,
                        help="your four predicted token counts for the lesson's strings, e.g. 4,6,8,2")
    parser.add_argument("--text", action="append", default=[], help="an extra string to tokenise; repeatable")
    parser.add_argument("--labbook", default=None, help="append one JSON line per run to this file")
    args = parser.parse_args()

    model_dir = Path(args.model).expanduser()
    if model_dir.is_dir() and not (model_dir / "tokenizer.json").exists():
        raise SystemExit(f"{model_dir} has no tokenizer.json; the download did not finish, or the path is wrong")

    guesses: list[int | None] = [None] * len(PREDICTIONS)
    if args.guess:
        parts = [p.strip() for p in args.guess.split(",")]
        if len(parts) != len(PREDICTIONS) or not all(p.isdigit() for p in parts):
            raise SystemExit(f"--guess needs {len(PREDICTIONS)} integers separated by commas, e.g. 4,6,8,2")
        guesses = [int(p) for p in parts]

    tokeniser = AutoTokenizer.from_pretrained(args.model)
    print(f"tokeniser class:            {type(tokeniser).__name__}")
    print(f"base vocabulary (BPE):      {tokeniser.vocab_size:,}   (tokeniser.vocab_size)")
    print(f"entries incl. added tokens: {len(tokeniser):,}   (len(tokeniser))")
    config_path = model_dir / "config.json"
    if config_path.exists():
        config_vocab = json.loads(config_path.read_text(encoding="utf-8")).get("vocab_size")
        if config_vocab:
            print(f"embedding rows (config):    {config_vocab:,}   (vocab_size in config.json; "
                  f"{config_vocab - len(tokeniser):,} rows no token can select)")
    print(f"model max length:           {tokeniser.model_max_length:,}")

    specials = []
    for name in ("bos_token", "eos_token", "pad_token", "unk_token"):
        token = getattr(tokeniser, name, None)
        token_id = getattr(tokeniser, f"{name}_id", None)
        print(f"  {name:10s} {token!r:18s} id {token_id}")
        specials.append({"name": name, "token": token, "id": token_id})
    print("  chat control tokens in this vocabulary:")
    controls = []
    for token in CONTROL_TOKENS:
        token_id = tokeniser.convert_tokens_to_ids(token)
        if token_id is not None and token_id != tokeniser.unk_token_id:
            print(f"    {token!r:16s} id {token_id}")
            controls.append({"token": token, "id": token_id})

    print("\n=== the four strings you predicted in the tokens lesson ===")
    results = [describe(tokeniser, label, text, guess)
               for (label, text), guess in zip(PREDICTIONS, guesses)]
    print("\n=== four kinds of text ===")
    results += [describe(tokeniser, label, text) for label, text in SAMPLES]
    results += [describe(tokeniser, f"custom-{i + 1}", text) for i, text in enumerate(args.text)]

    print("\nsummary")
    print(f"  {'sample':12s} {'tokens':>7s} {'guess':>6s} {'words':>7s} {'bytes':>7s} {'bytes/token':>12s} {'round-trip':>11s}")
    for r in results:
        per_token = r["utf8_bytes"] / max(r["tokens"], 1)
        guess = str(r["guess"]) if r["guess"] is not None else "-"
        print(f"  {r['label']:12s} {r['tokens']:7d} {guess:>6s} {r['words']:7d} {r['utf8_bytes']:7d} "
              f"{per_token:12.2f} {str(r['round_trip']):>11s}")

    if args.labbook:
        record = {
            "lab": "part-02/tokenise-samples",
            "model": args.model,
            "tokeniser": type(tokeniser).__name__,
            "base_vocab_size": tokeniser.vocab_size,
            "entries_with_added_tokens": len(tokeniser),
            "special_tokens": specials,
            "control_tokens": controls,
            "samples": results,
        }
        with Path(args.labbook).expanduser().open("a", encoding="utf-8") as fh:
            fh.write(json.dumps(record, ensure_ascii=False) + "\n")
        print(f"\nrecorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download tokenise-samples.py146 lines

The tokens lesson asked you to predict four token counts; the script prints those four strings first, with your guesses beside the answers if you pass them as --guess in the lesson’s order: Hello, world!, its Portuguese equivalent, the function signature, the two emoji. Write your four numbers down, then run:

RunnableAll tracks

tokenise the sample strings, with your predictions

python tokenise-samples.py --model ~/llm-course/models/qwen3-1.7b --guess 4,6,8,2 --labbook labbook.md

Replace 4,6,8,2 with your own guesses. Every id and piece below is exact for your download: the Qwen3-1.7B tokenizer.json is byte-identical to the one the author’s dry run used (same SHA-256), and the base repository’s smaller file produced the same ids for every string on this page. The 4 MB is serialisation: both files hold the same 151,643-entry vocabulary and the same 151,387 merges, but the instruct file writes each merge as a two-string list where the base file writes one "a b" string, pretty-printed over more lines (757,479 newlines against 303,281). The base file registers 22 added tokens against the instruct file’s 26, and tokenizer_config.json supplies the missing four (<tool_response>, </tool_response>, <think>, </think>) when Transformers loads it, so len(tokeniser) is 151,669 and <think> is id 151667 in both. Checked on the 0.6B pair, whose two files are byte-identical to the 1.7B pair’s.

Output — what you should see

tokeniser class:            Qwen2Tokenizer
base vocabulary (BPE):      151,643   (tokeniser.vocab_size)
entries incl. added tokens: 151,669   (len(tokeniser))
embedding rows (config):    151,936   (vocab_size in config.json; 267 rows no token can select)
model max length:           131,072
  bos_token  None               id None
  eos_token  '<|im_end|>'       id 151645
  pad_token  '<|endoftext|>'    id 151643
  unk_token  None               id None
  chat control tokens in this vocabulary:
    '<|endoftext|>'  id 151643
    '<|im_start|>'   id 151644
    '<|im_end|>'     id 151645
    '<think>'        id 151667
    '</think>'       id 151668

=== the four strings you predicted in the tokens lesson ===

[hello]  'Hello, world!'
  tokens    4   words    2   characters   13   utf-8 bytes   13   your guess    4
  ids    [9707, 11, 1879, 0]
  pieces ['Hello', ',', 'Ġworld', '!']
  tokens per word 2.00   bytes per token 3.25
  decode round-trip identical: True

[hello-pt]  'Olá, mundo!'
  tokens    5   words    2   characters   11   utf-8 bytes   12   your guess    6
  ids    [42719, 1953, 11, 28352, 0]
  pieces ['Ol', 'á', ',', 'Ġmundo', '!']
  pieces decoded one by one ['Ol', 'á', ',', ' mundo', '!']
  ...

[signature]  'def get_user_by_id(user_id: int):'
  tokens   10   words    3   characters   33   utf-8 bytes   33   your guess    8
  ids    [750, 633, 3317, 3710, 842, 4277, 842, 25, 526, 1648]
  pieces ['def', 'Ġget', '_user', '_by', '_id', '(user', '_id', ':', 'Ġint', '):']
  ...

[flag-rocket]  '🇧🇷🚀'
  tokens    3   words    1   characters    3   utf-8 bytes   12   your guess    2
  ids    [145340, 145070, 145836]
  pieces ['ðŁĩ§', 'ðŁĩ·', 'ðŁļĢ']
  pieces decoded one by one ['🇧', '🇷', '🚀']
  ...

=== four kinds of text ===

[english]  'The quick brown fox jumps over the lazy dog near the river bank.'
  tokens   14   words   13   characters   64   utf-8 bytes   64
  pieces ['The', 'Ġquick', 'Ġbrown', 'Ġfox', 'Ġjumps', 'Ġover', 'Ġthe', 'Ġlazy', 'Ġdog', 'Ġnear', 'Ġthe', 'Ġriver', 'Ġbank', '.']
  ...
[portuguese]  'A rápida raposa castanha salta sobre o cão preguiçoso perto da margem do rio.'
  tokens   25   words   14   characters   77   utf-8 bytes   80
  pieces ['A', 'Ġrápida', 'Ġrap', 'osa', 'Ġcast', 'an', 'ha', 'Ġsal', 'ta', 'Ġsobre', 'Ġo', 'Ġc', 'ão', 'Ġpre', 'gui', 'ç', 'oso', 'Ġp', 'erto', 'Ġda', 'Ġmarg', 'em', 'Ġdo', 'Ġrio', '.']
  ...
[code]  'def get_user_by_id(user_id: int) -> User | None:\n    return session.get(User, user_id)\n'
  tokens   24   words   10   characters   87   utf-8 bytes   87
  pieces ['def', 'Ġget', '_user', '_by', '_id', '(user', '_id', ':', 'Ġint', ')', 'Ġ->', 'ĠUser', 'Ġ|', 'ĠNone', ':Ċ', 'ĠĠĠ', 'Ġreturn', 'Ġsession', '.get', '(User', ',', 'Ġuser', '_id', ')Ċ']
  ...
[emoji]  '🇧🇷 🚀 ✨ 🧮'
  tokens   10   words    4   characters    8   utf-8 bytes   22
  pieces ['ðŁĩ§', 'ðŁĩ·', 'ĠðŁ', 'ļ', 'Ģ', 'Ġâľ', '¨', 'ĠðŁ', '§', '®']
  ...

summary
  sample        tokens  guess   words   bytes  bytes/token  round-trip
  hello              4      4       2      13         3.25        True
  hello-pt           5      6       2      12         2.40        True
  signature         10      8       3      33         3.30        True
  flag-rocket        3      2       1      12         4.00        True
  english           14      -      13      64         4.57        True
  portuguese        25      -      14      80         3.20        True
  code              24      -      10      87         3.62        True
  emoji             10      -       4      22         2.20        True

recorded in labbook.md

Six things to read out of this, five of which the tokens lesson predicted.

Three vocabulary sizes, all right. The merge process produced 151,643 entries; 26 added special tokens make 151,669; and config.json declares 151,936 embedding rows, 1,187 × 128, a width the matrix kernels like, so 267 rows are parameters no input can select. The parameters lesson counted the third figure, correctly, because that is the tensor’s shape.

Ġ is a space and Ċ a newline. The tokeniser is byte-level: it maps each of the 256 byte values to a printable character and runs BPE over those, so Ġworld and world are different tokens with different rows in the matrix you read in task 4. SentencePiece’s ▁ is the same idea in another family.

Non-ASCII text shows as its bytes. á is C3 A1 in UTF-8 and shows as á, hence 'Ġrápida'; a flag emoji, four bytes per regional-indicator symbol, shows as 'ðŁĩ§'. The “decoded one by one” line proves each piece holds the text it seems to; a piece holding part of a multi-byte character decodes alone to a replacement character, and the sequence still round-trips.

Portuguese costs 25 tokens where English costs 14 for the same sentence: rápida, castanha, preguiçoso and perto did not survive the merge process as whole words, so the same meaning is 79 percent more expensive in context window and, later, in KV cache. Compare languages by bytes per token, 3.20 against 4.57, rather than by raw counts.

Code is spiky. get_user_by_id is four tokens and the signature ten; return and session are one each; four spaces of indentation are one token, ĠĠĠ plus the space carried by Ġreturn. Task 6’s second prompt shows 1969 split into four single-digit tokens, the mechanical reason the tokens lesson gave for fragile arithmetic.

The round trip is lossless, True on every line, which is why an engine can stream tokens and reassemble the text without a table of exceptions. The control tokens at the top are ordinary entries in the same table with rows in the same matrix; task 7 turns them into a conversation, and the two eos_token_id values in the two config.json files, 151643 <|endoftext|> for the base and 151645 <|im_end|> for the instruct, are two of them.

✕Common mistake

tokeniser(text) adds special tokens by default in many models; the script passes add_special_tokens=False so that the counts are the text and nothing else. Qwen3 adds no beginning-of-sequence token either way; on a model that does, a count off by one at the start of every sample is this default.

Record in the notebook: the script’s JSON line holds every count; add the gap between your four guesses and the four answers, because that gap is the measurement.

6. One forward pass: probabilities and attention

RunnableAll tracks

next-token-and-attention.py

"""Run one forward pass: print the next-token distribution and one head's attention.

Purpose: show the two things the lessons describe but cannot show on a page - the
         probability the model assigns to every token in its vocabulary for the position
         after the prompt (with the effect of temperature on that distribution), and the
         attention weights of one head in one layer, checked for the causal mask.
Platform: all (cuda on Tracks S and N, ROCm on Track X, mps on Track M, or the CPU;
          the device is chosen automatically and printed)
Minimum memory: 8 GB (about 3.4 GB of weights for Qwen3-1.7B in BF16)
Assumes: a model directory downloaded with `hf download ... --local-dir <dir>`, and
         torch, transformers, safetensors and accelerate installed in the course
         environment from Part 1.

Usage: python next-token-and-attention.py --model ~/llm-course/models/qwen3-1.7b
           [--prompt "The capital of France is"] [--layer -1] [--head 0]
           [--top 10] [--dtype auto|float32|bfloat16|float16] [--labbook labbook.md]
"""
import argparse
import json
from pathlib import Path

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

TEMPERATURES = (0.5, 1.0, 1.5)


def pick_device() -> torch.device:
    if torch.cuda.is_available():
        return torch.device("cuda")
    mps = getattr(torch.backends, "mps", None)
    if mps is not None and mps.is_available():
        return torch.device("mps")
    return torch.device("cpu")


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--model", required=True, help="model directory or Hub id")
    parser.add_argument("--prompt", default="The capital of France is")
    parser.add_argument("--layer", type=int, default=-1, help="layer index; -1 is the last")
    parser.add_argument("--head", type=int, default=0, help="attention head index within that layer")
    parser.add_argument("--top", type=int, default=10, help="how many next tokens to print")
    parser.add_argument("--dtype", default="auto", choices=["auto", "float32", "bfloat16", "float16"])
    parser.add_argument("--max-print", type=int, default=24, help="print the full matrix only up to this many tokens")
    parser.add_argument("--labbook", default=None, help="append one JSON line per run to this file")
    args = parser.parse_args()

    model_dir = Path(args.model).expanduser()
    if model_dir.is_dir() and not (model_dir / "config.json").exists():
        raise SystemExit(f"{model_dir} has no config.json; the download did not finish, or the path is wrong")

    device = pick_device()
    if args.dtype == "auto":
        # bfloat16 is the precision the checkpoint was saved in and halves the weight
        # memory, which is what keeps this lab inside the 8 GB tier. If an operator is
        # missing for bfloat16 on your CPU, rerun with --dtype float32 and twice the memory.
        dtype = torch.bfloat16
    else:
        dtype = getattr(torch, args.dtype)
    print(f"device: {device}   dtype: {dtype}   torch {torch.__version__}")

    tokeniser = AutoTokenizer.from_pretrained(args.model)
    # The eager implementation computes the attention matrix explicitly, which is what
    # output_attentions=True returns; the optimised backends never build it.
    model = AutoModelForCausalLM.from_pretrained(
        args.model, dtype=dtype, attn_implementation="eager",
    ).to(device)
    model.eval()

    config = model.config
    groups = config.num_attention_heads // config.num_key_value_heads
    print(f"layers: {config.num_hidden_layers}   query heads: {config.num_attention_heads}   "
          f"key-value heads: {config.num_key_value_heads}   head_dim: {config.head_dim}   "
          f"vocab: {config.vocab_size:,}")
    print(f"weights once loaded: {sum(p.numel() for p in model.parameters()):,} parameters")

    inputs = tokeniser(args.prompt, return_tensors="pt").to(device)
    ids = inputs["input_ids"][0].tolist()
    pieces = tokeniser.convert_ids_to_tokens(ids)
    print(f"\nprompt: {args.prompt!r}")
    print(f"tokens ({len(ids)}): {pieces}")
    print(f"ids: {ids}")

    with torch.no_grad():
        out = model(**inputs, output_attentions=True)

    # ---- the next-token distribution -------------------------------------------------
    print(f"\nlogits: tensor shape {tuple(out.logits.shape)} = (batch, prompt tokens, vocabulary)")
    logits = out.logits[0, -1].float()
    probs = torch.softmax(logits, dim=-1)
    top = torch.topk(probs, args.top)
    print(f"\ntop {args.top} next tokens after {pieces[-1]!r} (softmax of the last row, temperature 1):")
    print(f"  {'rank':>4s} {'id':>8s} {'logit':>9s} {'probability':>12s}  token")
    top_records = []
    for rank, (p, token_id) in enumerate(zip(top.values.tolist(), top.indices.tolist()), start=1):
        token = tokeniser.convert_ids_to_tokens(token_id)
        print(f"  {rank:4d} {token_id:8d} {float(logits[token_id]):9.3f} {p:12.6f}  {token!r}")
        top_records.append({"rank": rank, "id": token_id, "token": token,
                            "logit": round(float(logits[token_id]), 3), "probability": round(p, 6)})
    covered = float(top.values.sum())
    print(f"  the other {config.vocab_size - args.top:,} tokens share {1 - covered:.6f} of the probability")

    # The same logits under three temperatures: softmax(logits / T).
    print(f"\ntemperature on the same logits (probability of the top token, and of the top {args.top} together):")
    temperature_records = []
    for temp in TEMPERATURES:
        p_t = torch.softmax(logits / temp, dim=-1)
        top_t = torch.topk(p_t, args.top)
        entropy = float(-(p_t * torch.log(p_t.clamp_min(1e-12))).sum())
        print(f"  T = {temp:3.1f}   top-1 {top_t.values[0].item():.4f}   top-{args.top} {top_t.values.sum().item():.4f}"
              f"   entropy {entropy:5.2f} nats")
        temperature_records.append({"temperature": temp, "top1": round(top_t.values[0].item(), 4),
                                    "topk": round(top_t.values.sum().item(), 4), "entropy_nats": round(entropy, 2)})

    # ---- one head's attention --------------------------------------------------------
    layer_index = args.layer if args.layer >= 0 else len(out.attentions) + args.layer
    if not 0 <= layer_index < len(out.attentions):
        raise SystemExit(f"--layer {args.layer} is outside 0..{len(out.attentions) - 1}")
    heads = out.attentions[layer_index].shape[1]
    if not 0 <= args.head < heads:
        raise SystemExit(f"--head {args.head} is outside 0..{heads - 1}")

    print(f"\nattentions: {len(out.attentions)} tensors (one per layer), each of shape "
          f"{tuple(out.attentions[layer_index].shape)} = (batch, heads, query position, key position)")
    attention = out.attentions[layer_index][0, args.head].float()
    print(f"layer {layer_index}, query head {args.head} (it reads key-value head {args.head // groups}, "
          f"shared by {groups} query heads): tensor shape {tuple(attention.shape)}")

    row_sums = attention.sum(dim=-1)
    above = attention.triu(diagonal=1)
    print(f"  row sums: min {row_sums.min().item():.4f}  max {row_sums.max().item():.4f}  "
          f"(one, within the rounding of {dtype})")
    print(f"  largest entry above the diagonal: {above.max().item():.6f}  (the causal mask; must be exactly 0)")

    last_row = attention[-1]
    order = torch.argsort(last_row, descending=True)
    print(f"\n  what the last token {pieces[-1]!r} attended to:")
    attended = []
    for pos in order[: min(5, len(ids))].tolist():
        weight = float(last_row[pos])
        print(f"    position {pos:3d}  weight {weight:7.4f}  {pieces[pos]!r}")
        attended.append({"position": pos, "weight": round(weight, 4), "token": pieces[pos]})

    if len(ids) <= args.max_print:
        print("\n  full matrix, rounded to two decimals (rows: query position; columns: key position):")
        header = "        " + " ".join(f"{i:5d}" for i in range(len(ids)))
        print(header)
        for i in range(len(ids)):
            row = " ".join(f"{float(attention[i, j]):5.2f}" for j in range(len(ids)))
            print(f"    {i:3d} {row}")
    else:
        print(f"\n  (matrix not printed: {len(ids)} tokens is more than --max-print {args.max_print})")

    if args.labbook:
        record = {
            "lab": "part-02/next-token-and-attention",
            "model": args.model, "device": str(device), "dtype": str(dtype),
            "torch": torch.__version__,
            "prompt": args.prompt, "prompt_tokens": len(ids),
            "logits_shape": list(out.logits.shape),
            "top_tokens": top_records,
            "temperature": temperature_records,
            "layer": layer_index, "head": args.head, "kv_head": args.head // groups,
            "attention_shape": list(attention.shape),
            "row_sum_min": round(row_sums.min().item(), 4), "row_sum_max": round(row_sums.max().item(), 4),
            "max_above_diagonal": above.max().item(),
            "last_token_attended": attended,
        }
        with Path(args.labbook).expanduser().open("a", encoding="utf-8") as fh:
            fh.write(json.dumps(record, ensure_ascii=False) + "\n")
        print(f"\nrecorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download next-token-and-attention.py176 lines

RunnableAll tracks

the next-token distribution and one head's attention

python next-token-and-attention.py \
  --model ~/llm-course/models/qwen3-1.7b \
  --prompt "The capital of France is" \
  --layer -1 --head 0 --top 10 \
  --labbook labbook.md

The shapes, ids and zeroes below are exact for Qwen3-1.7B; the logits, probabilities and attention weights are the author’s dry run on Qwen3-0.6B on a CPU in BF16 (2026-09-12), so that you know what a plausible run looks like; yours will differ in every one of those digits.

Output — what you should see

device: cuda   dtype: torch.bfloat16   torch 2.x.x
layers: 28   query heads: 16   key-value heads: 8   head_dim: 128   vocab: 151,936
weights once loaded: 1,720,574,976 parameters

prompt: 'The capital of France is'
tokens (5): ['The', 'Ġcapital', 'Ġof', 'ĠFrance', 'Ġis']
ids: [785, 6722, 315, 9625, 374]

logits: tensor shape (1, 5, 151936) = (batch, prompt tokens, vocabulary)

top 10 next tokens after 'Ġis' (softmax of the last row, temperature 1):
  rank       id     logit  probability  token
     1    12095    17.500     0.659651  'ĠParis'
     2     7407    14.312     0.027227  'Ġlocated'
     3      279    14.062     0.021204  'Ġthe'
     4    30743    13.750     0.015514  'Ġ____'
   ...
  the other 151,926 tokens share 0.205525 of the probability

temperature on the same logits (probability of the top token, and of the top 10 together):
  T = 0.5   top-1 0.9933   top-10 0.9986   entropy  0.06 nats
  T = 1.0   top-1 0.6597   top-10 0.7945   entropy  2.29 nats
  T = 1.5   top-1 0.1563   top-10 0.2675   entropy  7.10 nats

attentions: 28 tensors (one per layer), each of shape (1, 16, 5, 5) = (batch, heads, query position, key position)
layer 27, query head 0 (it reads key-value head 0, shared by 2 query heads): tensor shape (5, 5)
  row sums: min 0.9986  max 1.0013  (one, within the rounding of torch.bfloat16)
  largest entry above the diagonal: 0.000000  (the causal mask; must be exactly 0)

  what the last token 'Ġis' attended to:
    position   0  weight  0.8672  'The'
    position   4  weight  0.0918  'Ġis'
    position   3  weight  0.0254  'ĠFrance'
    ...

  full matrix, rounded to two decimals (rows: query position; columns: key position):
            0     1     2     3     4
      0  1.00  0.00  0.00  0.00  0.00
      1  0.98  0.02  0.00  0.00  0.00
      2  0.96  0.02  0.02  0.00  0.00
      3  0.97  0.00  0.01  0.02  0.00
      4  0.87  0.00  0.01  0.03  0.09

recorded in labbook.md

A logit for every vocabulary entry at every position. The Transformers documentation gives logits the shape (batch_size, sequence_length, config.vocab_size), “scores for each vocabulary token before SoftMax”: (1, 5, 151936) here, of which the script uses the last row. Those scores are the last hidden state, width 2048, multiplied by the embedding matrix you listed in task 4, because the head is tied; the other four rows are the predictions for positions 2 to 5, computed in the same pass, which is what makes next-token prediction a training signal at every position at once.

Softmax turns scores into a distribution, and temperature divides the scores. With z the logits and T the temperature:

p(token i) = exp(z_i / T) / sum over all j of exp(z_j / T)

At T = 1 the top token took 0.66 of the probability in the dry run and the other 151,926 entries shared 0.21: even with one obvious answer, the output is a distribution, not an answer. Dividing every logit by 0.5 doubles the gaps and the top share went to 0.99; dividing by 1.5 shrinks them and it fell to 0.16 while the entropy rose from 2.29 to 7.10 nats. That is all temperature is; the next-token lesson’s sampling rules decide which entries to keep before drawing, and the model card’s recommended 0.6 in thinking mode and 0.7 otherwise is this divisor.

The model was loaded with eager attention. The script passes attn_implementation="eager" because, as the attention-backends documentation states, “basic attention scales poorly because it materializes the full attention matrix in memory” while “optimized implementations rearrange the math to reduce memory traffic”: there is no matrix for them to hand back. Eager builds it, which is fine for five tokens.

The attention tensor is square and triangular, and the script checks it. The documentation gives attentions as one tensor per layer of shape (batch_size, num_heads, sequence_length, sequence_length), “attentions weights after the attention softmax”: 28 tensors of (1, 16, 5, 5). Query head 0 reads key-value head 0, because 8 key-value heads serve 16 query heads two each, the grouped-query arrangement the attention lesson described and the KV-cache arithmetic later in the course depends on. Every row is one softmax and sums to one within BF16 rounding, and the largest entry above the diagonal is exactly zero: the mask sets those scores to minus infinity before the softmax, and exp(−∞) is zero, not small.

Now a longer prompt, a different head and a different layer:

RunnableAll tracks

a longer prompt, and a different head in the first layer

python next-token-and-attention.py \
  --model ~/llm-course/models/qwen3-1.7b \
  --prompt "In 1969 the first crewed mission landed on the Moon, and the commander was" \
  --layer 0 --head 3 --top 5 \
  --labbook labbook.md

Output — what you should see

prompt: 'In 1969 the first crewed mission landed on the Moon, and the commander was'
tokens (20): ['In', 'Ġ', '1', '9', '6', '9', 'Ġthe', 'Ġfirst', 'Ġcrew', 'ed', 'Ġmission', 'Ġlanded', 'Ġon', 'Ġthe', 'ĠMoon', ',', 'Ġand', 'Ġthe', 'Ġcommander', 'Ġwas']
ids: [641, 220, 16, 24, 21, 24, 279, 1156, 13627, 291, 8954, 26120, 389, 279, 17330, 11, 323, 279, 27994, 572]

logits: tensor shape (1, 20, 151936) = (batch, prompt tokens, vocabulary)
...
attentions: 28 tensors (one per layer), each of shape (1, 16, 20, 20) = (batch, heads, query position, key position)
layer 0, query head 3 (it reads key-value head 1, shared by 2 query heads): tensor shape (20, 20)
  row sums: min 0.99xx  max 1.00xx  (one, within the rounding of torch.bfloat16)
  largest entry above the diagonal: 0.000000  (the causal mask; must be exactly 0)
...

1969 is five tokens, a space and four digits. The matrix is (20, 20): 400 entries per head per layer instead of 25, sixteen times the work for four times the tokens, the attention lesson’s cost argument in numbers. In the dry run this first-layer head put 0.99 of the last token’s weight on itself where the last-layer head spread its weight over the prompt; early layers often attend locally, but heads are not interpretable units, and one that seems to “track the subject” on one prompt does not on the next. What is reliably true is the shape, the zeroes above the diagonal and the row sums. If the top token is nonsense, check that --model points at a directory whose listing matched task 2; if either check line is off, see Troubleshooting.

Record in the notebook: the script’s JSON line holds the top tokens, the temperature table and the two checks; add the tensor shapes in words and which key-value head your query head used.

7. Base and instruct, same question

RunnableAll tracks

base-vs-instruct.py

"""Give the same prompt to a base checkpoint and to its instruction-tuned sibling.

Purpose: show that a chat assistant is the same next-token loop with a template around it,
         by running one question through a base model as raw text and through the
         instruction-tuned model through its own chat template, printing the template's
         control tokens, the tokens it adds, and how many generated tokens went into the
         thinking block and the answer.
Platform: all (cuda on Tracks S and N, ROCm on Track X, mps on Track M, or the CPU;
          the device is chosen automatically and printed)
Minimum memory: 8 GB (one checkpoint of about 3.4 GB in BF16 is resident at a time)
Assumes: both checkpoints downloaded with `hf download ... --local-dir <dir>`, and torch,
         transformers and accelerate installed in the course environment from Part 1.
         The two models are loaded one at a time and released, so only one is resident.

Usage: python base-vs-instruct.py --base ~/llm-course/models/qwen3-1.7b-base
                                  --instruct ~/llm-course/models/qwen3-1.7b
           [--prompt "..."] [--max-new-tokens 80] [--no-thinking] [--labbook labbook.md]
"""
import argparse
import gc
import json
import time
from pathlib import Path

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

THINK_START = "<think>"
THINK_END = "</think>"


def pick_device() -> torch.device:
    if torch.cuda.is_available():
        return torch.device("cuda")
    mps = getattr(torch.backends, "mps", None)
    if mps is not None and mps.is_available():
        return torch.device("mps")
    return torch.device("cpu")


def release(model) -> None:
    """Drop a model and give the memory back before loading the next one."""
    del model
    gc.collect()
    if torch.cuda.is_available():
        torch.cuda.empty_cache()


def check_dir(path: str, label: str) -> None:
    p = Path(path).expanduser()
    if p.is_dir() and not (p / "config.json").exists():
        raise SystemExit(f"--{label} {p} has no config.json; the download did not finish, or the path is wrong")


def run(path: str, prompt_ids, device: torch.device, dtype, max_new_tokens: int, tokeniser) -> dict:
    """Load a checkpoint, generate greedily from prepared ids, print and release."""
    model = AutoModelForCausalLM.from_pretrained(path, dtype=dtype).to(device)
    model.eval()
    started = time.perf_counter()
    with torch.no_grad():
        out = model.generate(
            **prompt_ids.to(device),
            do_sample=False,
            max_new_tokens=max_new_tokens,
            pad_token_id=tokeniser.pad_token_id or tokeniser.eos_token_id,
        )
    seconds = time.perf_counter() - started
    prompt_len = prompt_ids["input_ids"].shape[1]
    generated = out[0, prompt_len:].tolist()
    stopped_on_eos = len(generated) < max_new_tokens or generated[-1] in {tokeniser.eos_token_id, tokeniser.pad_token_id}
    release(model)
    return {"ids": generated, "seconds": seconds, "stopped_on_eos": stopped_on_eos,
            "text": tokeniser.decode(generated, skip_special_tokens=True)}


def split_thinking(generated: list[int], tokeniser) -> tuple[str, str, int, int, bool]:
    """Split a Qwen3 reply at the </think> token, as the model card describes.

    Returns (thinking, answer, thinking_tokens, answer_tokens, closed). When the reply
    opened a <think> block that never closed, the whole reply is thinking and closed is
    False: the budget ran out inside the block.
    """
    think_start_id = tokeniser.convert_tokens_to_ids(THINK_START)
    think_end_id = tokeniser.convert_tokens_to_ids(THINK_END)
    if think_end_id is not None and think_end_id in generated:
        cut = len(generated) - generated[::-1].index(think_end_id)
        thinking = tokeniser.decode(generated[:cut], skip_special_tokens=True).strip("\n")
        answer = tokeniser.decode(generated[cut:], skip_special_tokens=True).strip("\n")
        return thinking, answer, cut, len(generated) - cut, True
    if think_start_id is not None and think_start_id in generated:
        thinking = tokeniser.decode(generated, skip_special_tokens=True).strip("\n")
        return thinking, "", len(generated), 0, False
    return "", tokeniser.decode(generated, skip_special_tokens=True).strip("\n"), 0, len(generated), True


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("--base", required=True, help="base checkpoint directory or Hub id")
    parser.add_argument("--instruct", required=True, help="instruction-tuned checkpoint directory or Hub id")
    parser.add_argument("--prompt", default="What is the capital of Brazil?")
    parser.add_argument("--max-new-tokens", type=int, default=80)
    parser.add_argument("--no-thinking", action="store_true",
                        help="pass enable_thinking=False to the chat template (a Qwen3 template option)")
    parser.add_argument("--dtype", default="auto", choices=["auto", "float32", "bfloat16", "float16"])
    parser.add_argument("--labbook", default=None, help="append one JSON line per run to this file")
    args = parser.parse_args()
    check_dir(args.base, "base")
    check_dir(args.instruct, "instruct")

    device = pick_device()
    # bfloat16 is how these checkpoints were saved; --dtype float32 needs twice the memory.
    dtype = torch.bfloat16 if args.dtype == "auto" else getattr(torch, args.dtype)
    print(f"device: {device}   dtype: {dtype}   torch {torch.__version__}   greedy decoding (do_sample=False)")

    # ---- the base model sees the prompt as ordinary text -----------------------------
    base_tokeniser = AutoTokenizer.from_pretrained(args.base)
    base_inputs = base_tokeniser(args.prompt, return_tensors="pt")
    base_len = int(base_inputs["input_ids"].shape[1])
    print(f"\n=== base: {args.base}")
    print(f"input as sent  : {args.prompt!r}")
    print(f"input tokens   : {base_len}  {base_tokeniser.convert_ids_to_tokens(base_inputs['input_ids'][0].tolist())}")
    base = run(args.base, base_inputs, device, dtype, args.max_new_tokens, base_tokeniser)
    print(f"generated      : {len(base['ids'])} tokens in {base['seconds']:.1f} s; stopped on end-of-text: {base['stopped_on_eos']}")
    print(f"continuation   : {base['text']!r}")

    # ---- the instruction-tuned model gets the same question through its template ------
    chat_tokeniser = AutoTokenizer.from_pretrained(args.instruct)
    messages = [{"role": "user", "content": args.prompt}]
    template_kwargs = {"add_generation_prompt": True}
    if args.no_thinking:
        template_kwargs["enable_thinking"] = False
    rendered = chat_tokeniser.apply_chat_template(messages, tokenize=False, **template_kwargs)
    chat_inputs = chat_tokeniser.apply_chat_template(
        messages, tokenize=True, return_dict=True, return_tensors="pt", **template_kwargs,
    )
    chat_ids = chat_inputs["input_ids"][0].tolist()
    print(f"\n=== instruct: {args.instruct}")
    print(f"input as sent  : the template's own string, control tokens included"
          f"{' (enable_thinking=False)' if args.no_thinking else ''}")
    print("---")
    print(rendered, end="")
    print("---")
    print(f"input tokens   : {len(chat_ids)}  {chat_tokeniser.convert_ids_to_tokens(chat_ids)}")
    print(f"template adds  : {len(chat_ids) - base_len} tokens around the {base_len} of the question")
    chat = run(args.instruct, chat_inputs, device, dtype, args.max_new_tokens, chat_tokeniser)
    thinking, answer, n_think, n_answer, closed = split_thinking(chat["ids"], chat_tokeniser)
    print(f"generated      : {len(chat['ids'])} tokens in {chat['seconds']:.1f} s; stopped on end-of-turn: {chat['stopped_on_eos']}")
    print(f"thinking block : {n_think} tokens{'' if closed else ' (never closed: the budget ran out inside it)'}  {thinking!r}")
    print(f"answer         : {n_answer} tokens  {answer!r}")

    if args.labbook:
        record = {
            "lab": "part-02/base-vs-instruct",
            "base": args.base, "instruct": args.instruct,
            "device": str(device), "dtype": str(dtype), "torch": torch.__version__,
            "prompt": args.prompt, "max_new_tokens": args.max_new_tokens,
            "enable_thinking": not args.no_thinking,
            "base_input_tokens": base_len,
            "chat_input_tokens": len(chat_ids),
            "template_overhead_tokens": len(chat_ids) - base_len,
            "base_generated_tokens": len(base["ids"]), "base_seconds": round(base["seconds"], 1),
            "base_completion": base["text"],
            "chat_generated_tokens": len(chat["ids"]), "chat_seconds": round(chat["seconds"], 1),
            "chat_thinking_tokens": n_think, "chat_thinking_closed": closed, "chat_answer_tokens": n_answer,
            "chat_thinking": thinking, "chat_answer": answer,
        }
        with Path(args.labbook).expanduser().open("a", encoding="utf-8") as fh:
            fh.write(json.dumps(record, ensure_ascii=False) + "\n")
        print(f"\nrecorded in {args.labbook}")


if __name__ == "__main__":
    main()

Download base-vs-instruct.py173 lines

The script sends one question to the base checkpoint as plain text and to the instruct checkpoint through its chat template.

RunnableAll tracks

the same prompt through both checkpoints

python base-vs-instruct.py \
  --base ~/llm-course/models/qwen3-1.7b-base \
  --instruct ~/llm-course/models/qwen3-1.7b \
  --prompt "What is the capital of Brazil?" \
  --max-new-tokens 80 \
  --labbook labbook.md

The token counts and the rendered template below are exact for Qwen3-1.7B; the generated texts and the seconds are the author’s 0.6B dry run on a CPU (2026-09-12), and yours will differ in wording, though almost certainly not in kind.

Output — what you should see

device: cuda   dtype: torch.bfloat16   torch 2.x.x   greedy decoding (do_sample=False)

=== base: /home/you/llm-course/models/qwen3-1.7b-base
input as sent  : 'What is the capital of Brazil?'
input tokens   : 7  ['What', 'Ġis', 'Ġthe', 'Ġcapital', 'Ġof', 'ĠBrazil', '?']
generated      : 80 tokens in xx.x s; stopped on end-of-text: False
continuation   : ' What is the capital of the United States? What is the capital of France? What is the capital of Germany? ...'

=== instruct: /home/you/llm-course/models/qwen3-1.7b
input as sent  : the template's own string, control tokens included
---
<|im_start|>user
What is the capital of Brazil?<|im_end|>
<|im_start|>assistant
---
input tokens   : 15  ['<|im_start|>', 'user', 'Ċ', 'What', 'Ġis', 'Ġthe', 'Ġcapital', 'Ġof', 'ĠBrazil', '?', '<|im_end|>', 'Ċ', '<|im_start|>', 'assistant', 'Ċ']
template adds  : 8 tokens around the 7 of the question
generated      : 80 tokens in xx.x s; stopped on end-of-turn: False
thinking block : 80 tokens (never closed: the budget ran out inside it)  "<think>\nOkay, the user is asking for the capital of Brazil. Let me think. ..."
answer         : 0 tokens  ''

recorded in labbook.md

The base model continued the text. Given seven tokens that look like a quiz question, the likeliest continuation under greedy decoding was another quiz question, then another; in the dry run it produced eleven and never emitted its end-of-text token. That is the pretraining objective doing what it was trained to do, and the chat-template documentation says it in one sentence: “All causal LMs, whether chat-trained or not, continue a sequence of tokens.”

The instruct model received a different input. Between the --- lines is the string the model saw: the question wrapped in <|im_start|>user and <|im_end|>, then <|im_start|>assistant and a newline, which the documentation’s add_generation_prompt argument appends so that the model “will correctly write a response” instead of continuing the user’s message. Those eight tokens are the template’s overhead on every request; with You are a helpful assistant. as a system message the same question is 26 tokens, and Part 9 measures what each costs at prefill.

Post-training made a reply the likely continuation of that pattern. Same architecture, same tokeniser, same 1,409,410,048 non-embedding parameters; different values in them, because the model was trained further on text in which <|im_start|>assistant is followed by an answer, and in Qwen3’s case by a <think> block first: the card describes thinking mode as on by default, and in the dry run all eighty tokens went into the block. The script splits the reply at the </think> token, id 151668, as the card’s quickstart does, and reports whether the block closed; eighty tokens is a budget, not a verdict.

Now with thinking off. The card documents enable_thinking=False as an argument to apply_chat_template, and the script passes it for you:

RunnableAll tracks

the same question with thinking turned off

python base-vs-instruct.py \
  --base ~/llm-course/models/qwen3-1.7b-base \
  --instruct ~/llm-course/models/qwen3-1.7b \
  --prompt "What is the capital of Brazil?" \
  --no-thinking --max-new-tokens 80 \
  --labbook labbook.md

Output — what you should see

=== instruct: /home/you/llm-course/models/qwen3-1.7b
input as sent  : the template's own string, control tokens included (enable_thinking=False)
---
<|im_start|>user
What is the capital of Brazil?<|im_end|>
<|im_start|>assistant
<think>

</think>

---
input tokens   : 19  ['<|im_start|>', 'user', 'Ċ', 'What', 'Ġis', 'Ġthe', 'Ġcapital', 'Ġof', 'ĠBrazil', '?', '<|im_end|>', 'Ċ', '<|im_start|>', 'assistant', 'Ċ', '<think>', 'ĊĊ', '</think>', 'ĊĊ']
template adds  : 12 tokens around the 7 of the question
generated      : x tokens in x.x s; stopped on end-of-turn: True
thinking block : 0 tokens  ''
answer         : x tokens  'The capital of Brazil is Brasília.'

The switch is in the prompt: the template writes an empty thinking block, <think>, two newlines, </think>, two newlines, four more tokens, and the model continues past a closed block with the answer. Nothing in the weights changed. The soft switches the card describes, /think and /no_think typed into a user message, are the same idea; Part 3’s post-training lesson explains what the block is for, and Part 10’s lesson on prompting, chat templates and thinking modes owns the switches.

Then thinking on again, with a budget the block can finish in:

RunnableAll tracks

thinking on, with a budget that lets the block close

python base-vs-instruct.py \
  --base ~/llm-course/models/qwen3-1.7b-base \
  --instruct ~/llm-course/models/qwen3-1.7b \
  --prompt "What is the capital of Brazil?" \
  --max-new-tokens 300 \
  --labbook labbook.md

Output — what you should see

...
generated      : xxx tokens in xx.x s; stopped on end-of-turn: True
thinking block : xxx tokens  "<think>\nOkay, the user is asking for the capital of Brazil. ... \n</think>"
answer         : xx tokens  'The capital of Brazil is Brasília.'

Record how many tokens the block took to close: that number, the reasoning budget, is what every later part that touches reasoning models has to plan for. If it still reports “never closed”, raise the budget again rather than concluding anything about the answer.

⌚Version note

The script decodes greedily so that a rerun gives the same text; the Transformers documentation describes greedy search as “the default decoding strategy” that “selects the next most likely token at each step”. The Qwen3-1.7B model card (read 2026-09-12) says the opposite for real use: “DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions”, recommending sampling at 0.6 in thinking mode and 0.7 otherwise. Greedy is for measurement, the card’s settings are for answers, and the base model’s eleven questions are the “endless repetitions” the card warns of, seen live.

Record in the notebook: the script’s JSON lines hold both inputs’ token counts, the overhead, the generated counts and the texts; add the number of tokens the thinking block took to close.

8. Track M only: the same generation in MLX

MLX is Apple’s array framework and the native path on Apple silicon; Parts 8 and 13 lean on it. Its README gives the Python API as load and generate from mlx_lm and a command-line entry point, mlx_lm.generate, whose --model option its source describes as “the path to the local model directory or Hugging Face repo”. Point it at the directory from task 2:

RunnableTrack M · Apple silicon

generate with the MLX command-line entry point, thinking on

mlx_lm.generate --model ~/llm-course/models/qwen3-1.7b --prompt "What is the capital of Brazil?" --max-tokens 300

Output — what you should see

==========
<think>
Okay, the user is asking for the capital of Brazil. ...
</think>

The capital of Brazil is Brasília.
==========
Prompt: 15 tokens, x.xxx tokens-per-sec
Generation: xxx tokens, x.xxx tokens-per-sec
Peak memory: x.xxx GB

Fifteen prompt tokens: the same template, because mlx_lm.generate applies the tokeniser’s chat template by default whenever one exists and passes any JSON given to --chat-template-config through to it, which is how thinking is switched off here:

RunnableTrack M · Apple silicon

thinking off, through the template option

mlx_lm.generate --model ~/llm-course/models/qwen3-1.7b --prompt "What is the capital of Brazil?" --max-tokens 300 --chat-template-config '{"enable_thinking": false}'

Output — what you should see

==========
The capital of Brazil is Brasília.
==========
Prompt: 19 tokens, x.xxx tokens-per-sec
Generation: x tokens, x.xxx tokens-per-sec
Peak memory: x.xxx GB

Nineteen tokens, the four of the empty block added, as in task 7. The base model needs one more flag, because its repository also ships a chat template that MLX would otherwise apply:

RunnableTrack M · Apple silicon

the base checkpoint, as raw text

mlx_lm.generate --model ~/llm-course/models/qwen3-1.7b-base --prompt "What is the capital of Brazil?" --max-tokens 60 --ignore-chat-template

Output — what you should see

==========
What is the capital of the United States? What is the capital of France? ...
==========
Prompt: 7 tokens, x.xxx tokens-per-sec
Generation: 60 tokens, x.xxx tokens-per-sec
Peak memory: x.xxx GB

Seven tokens, and the same kind of continuation. The three command shapes were exercised with mlx-lm 0.31.3 on its CPU-only build against the 0.6B directories; the Metal path is what the validation pass will record. Record the three Generation: lines next to the Transformers ones; Part 8 makes the comparison properly.

Reconcile the artefacts before interpreting model behaviour

Complete the configuration and tensor inspection before the generation comparison. This lets you separate a mistaken model identity from a behavioural difference between base and instruct models.

1. Record the downloaded repository and revision for each checkpoint. Confirm every shard named by the index exists before counting tensors.
2. Compare the tensor shapes with the configuration. Account for tied embeddings rather than adding the same shared parameter twice to your conceptual parameter count.
3. Save token IDs and decoded text for the same input under both tokenisers. If they differ, explain the consequence for a supposedly identical token budget.
4. Inspect the actual prompt format for generation. A raw continuation and a templated chat request ask different questions; label both if you run both.
5. Retain the next-token probabilities and attention output from one small forward pass. Check that masked future positions do not contribute before interpreting an attention plot.

If the configuration inspection works but weights cannot load, preserve that partial result and the first loader error. Reduce the workload using the lesson’s stated path instead of silently switching checkpoints. Your completed notebook should connect a named artefact, a tensor shape, a tokenisation observation and a generated response, with enough detail to reproduce each one.

Validation

You are done when every row passes.

Check	Command	Pass means
Both repositories are complete	ls -A ~/llm-course/models/qwen3-1.7b ~/llm-course/models/qwen3-1.7b-base	Twelve and ten entries including .gitattributes, plus the .cache directory the CLI adds, with config.json, tokenizer.json and the safetensors sizes from the table at the top
The count reconciles	python list-tensors.py --model ~/llm-course/models/qwen3-1.7b --layer 0	match: True twice, parameters once loaded: 1,720,574,976, lm_head.weight stored in the file: True
The base has no stored head	python list-tensors.py --model ~/llm-course/models/qwen3-1.7b-base	310 tensors and lm_head.weight stored in the file: False
Tokenisation is lossless and matches	python tokenise-samples.py --model ~/llm-course/models/qwen3-1.7b	True in every round-trip column, 14 tokens for English, 25 for Portuguese
The forward pass has the right shapes	task 6’s first command	logits: tensor shape (1, 5, 151936), 28 attention tensors of (1, 16, 5, 5)
The causal mask is exact	the same run	largest entry above the diagonal: 0.000000, row sums within 0.01 of one
Temperature behaves	the same run	top-1 at T = 0.5 above top-1 at T = 1.0 above top-1 at T = 1.5
The template is what you think	task 7’s first command	template adds : 8 tokens, the rendered string between the --- lines
The two checkpoints differ in kind	the same run	The base continues; the instruct replies or thinks; --no-thinking adds 4 tokens and an answer arrives
The notebook was written	grep -c '"lab": "part-02' ~/llm-course/labbook.md	At least 6 (two listings, one tokenisation, two forward passes, one or more comparisons)

RunnableAll tracks

confirm the notebook was written

grep -c '"lab": "part-02' ~/llm-course/labbook.md

Expected outcome

Two model directories you can point any tool at, and at least six recorded runs in the lab notebook whose numbers you derived rather than read, the 311,164,928 that the instruct repository stores twice among them. From here on the course assumes you know what is in a checkpoint, because you have listed one and added it up.

Troubleshooting

Symptom, cause, fix. Error texts are quoted from the documentation or from the scripts, and say so where a message may differ by version.

Symptom	Cause	Fix
ModuleNotFoundError: No module named 'transformers' in preflight 1	The install ran in another environment, or the environment is not activated	source ~/llm-course/.venv/bin/activate, then your track’s install line; on Track S, the pip install inside the container
bash: .venv/bin/activate: No such file or directory on Track S	You are inside the NGC container, which has no uv environment	Skip the line, as the Spark tab says; python and hf are the container’s own
hf: command not found	The hf executable comes with huggingface_hub, and the environment is not active	Activate, then uv pip install huggingface_hub if it is still missing
Error: Not logged in from hf auth whoami	You have not run task 1	Nothing, for this lab; the repositories are public
Warning: You are sending unauthenticated requests to the HF Hub...	Same	Same; sign in once if you want the higher rate limits the warning mentions
The download stops, or httpx.TimeoutException: ... Read timed out. (read timeout=10)	A slow or interrupted link; the documentation gives the default timeout as 10	Run the same hf download again; it resumes from the .cache/huggingface/ metadata. On a slow link, export HF_HUB_DOWNLOAD_TIMEOUT=30 first, as the documentation describes
no *.safetensors files in ...; the download did not finish, or the path is wrong	The script found the directory but no weights	Compare ls -A with task 2’s listing; rerun the download
KeyError: 'qwen3' when a script loads the model	Transformers older than 4.51.0, which the model card names as the minimum	uv pip install --upgrade transformers in the environment, then preflight 1 again
RuntimeError naming BFloat16 and an unimplemented operator	The CPU build lacks a BF16 kernel for one operation	Rerun with --dtype float32, which needs about 6.9 GB for the weights, or --dtype float16 on an accelerator that prefers it
Out of memory on an 8 GB machine	Two models resident, or another application holding memory	Run one script at a time and keep the default BF16; if base-vs-instruct.py still fails, give the same path to --base and --instruct and compare across two runs, or switch to the 0.6B pair from Requirements
CUDA out of memory on a small GPU (Track N)	The 3.44 GB of weights plus the CUDA context do not fit the card	Prefix the command with CUDA_VISIBLE_DEVICES="" to run it on the CPU; every count and shape is the same
out.attentions is None, or an error names the attention implementation	The model was loaded without attn_implementation="eager"	The script sets it; in a notebook of your own, put it back: the optimised backends never build the matrix output_attentions=True returns
Row sums far from one, or a non-zero above the diagonal	You are reading a tensor other than the attention weights, or edited the script	Rerun the unmodified script; both check lines come from the tensor the matrix is printed from
mps is selected but the run is slower than the CPU	For a model this small, transfer and launch overheads can dominate	Not a fault; note it, and Part 5 measures the same work on both properly
The instruct model produces only a thinking block and no answer	The reply ran out of --max-new-tokens inside the block, which the script reports as “never closed”	Raise --max-new-tokens to 300, or pass --no-thinking
--guess needs 4 integers separated by commas, e.g. 4,6,8,2	The lesson asked for four predictions; the option wants exactly four	Pass four integers, or omit --guess
Track M: mlx_lm.generate on the base model produces a chat-style reply	MLX applied the base repository’s chat template	Add --ignore-chat-template, as task 8 shows
A .cache/huggingface directory appears inside a model directory	Expected: the CLI’s metadata about the downloaded files	Leave it; it is what makes a rerun of the download cheap

Cleanup

Keep the environment, the notebook and the instruct directory: Part 4’s lab reads its config.json from ~/llm-course/models/qwen3-1.7b/, and Qwen3-1.7B returns as the student in Part 15’s distillation labs. If you need 3.4 GB back now, delete the base checkpoint; task 2’s command re-creates it.

⚠Warning

Deleting a model directory is irreversible and re-downloading it costs the full 3.5 GB. Check the path before you type it, and never point a recursive delete at ~/llm-course itself, which holds the notebook the capstone in Part 28 is written from.

RunnableAll tracks

optional: reclaim the disk taken by the base checkpoint

rm -rf ~/llm-course/models/qwen3-1.7b-base

Task 8 made no second copy: MLX read the same directory. count-from-config.py and embedding-row.py can stay in ~/llm-course or go; nothing later depends on them.

What you learned

• A checkpoint is a file format, and you read one without loading it: eight bytes of length, a 35,648-byte JSON header naming 310 tensors with their dtypes and shapes, then 3,441,149,952 bytes of BF16 data; a second shard of 120 bytes of header and one tensor.
• Published parameter counts are arithmetic you can redo: eleven shapes give 50,336,000 per block; 28 blocks, a final norm and a 151,936 × 2,048 embedding give 1,720,574,976, the card’s 1.7B; the repository holds 311,164,928 more because it stores the tied output head, which is why the download is 4.06 GB and the memory 3.44 GB.
• The tokeniser decides what text costs, and one tokeniser serves both checkpoints: four tokens for Hello, world!, five in Portuguese, ten for a signature, three for two emoji; 14 against 25 for one sentence in two languages; 1969 as four digits; a lossless round trip; three vocabulary sizes, each right for its purpose; a 4 MB file-size difference that is serialisation.
• The output is 151,936 scores per position, and temperature is a divisor on them: the top token’s share rose at T = 0.5 and collapsed at T = 1.5 on the same scores.
• Attention is an n by n matrix per head per layer, with exact zeroes above the diagonal: 28 tensors of (1, 16, 5, 5), then (1, 16, 20, 20); rows summing to one; query head 0 reading key-value head 0. This is the object whose growth makes long context expensive.
• Chat is a template around next-token prediction: eight control tokens around seven of question, four more to close an empty thinking block, a base model that continued with eleven more questions and an instruct model that replied, on the same architecture and tokeniser, because of what the later training made likely.

Check your understanding

Question 1. list-tensors.py read a 4 GB checkpoint and finished in well under a second. How?

• It memory-maps the weights and lets the operating system page them in lazily
• It reads the 8-byte length prefix and the JSON header, which name every tensor with its dtype, shape and offsets, and never touches the byte buffer
• It reads the model card and model.safetensors.index.json instead of the weights
• It samples a few tensors and extrapolates the total

Show the answer and why

Answer: It reads the 8-byte length prefix and the JSON header, which name every tensor with its dtype, shape and offsets, and never touches the byte buffer

The safetensors specification puts a JSON header between the length prefix and the data; for Qwen3-1.7B that header is 35,648 bytes for 310 tensors. The index JSON names shards but not shapes, so it could not have produced the shape column.

Question 2. Your listing of the instruct repository shows 2,031,739,904 parameters in the file, but the card says 1.7B and config.json says tie_word_embeddings is true. Which statement is correct?

• The card is wrong; the model has 2.03B parameters
• The file stores lm_head.weight, a second copy of the 151,936 × 2,048 embedding matrix, which the loader ties to the embedding; 1,720,574,976 parameters are resident once loaded
• The extra parameters are the KV cache, saved with the weights
• BF16 stores each parameter twice for precision

Show the answer and why

Answer: The file stores lm_head.weight, a second copy of the 151,936 × 2,048 embedding matrix, which the loader ties to the embedding; 1,720,574,976 parameters are resident once loaded

2,031,739,904 − 1,720,574,976 = 311,164,928 = 151,936 × 2,048, and the second shard holds exactly that one tensor. The base repository, with the same architecture and the same tie setting, stores 310 tensors and no lm_head.weight at all, so the difference is packaging, not architecture.

Question 3. The prompt grows from 5 tokens to 20. By what factor does the number of entries in one head's attention matrix grow, and what does the causal mask do to them?

• Four times; the mask zeroes the lower triangle
• Sixteen times, from 25 to 400 entries; the mask sets every entry above the diagonal to exactly zero, so a position never attends to a later one
• Four times; the mask has no effect on a prompt this short
• Sixteen times; the mask makes the entries above the diagonal very small but non-zero

Show the answer and why

Answer: Sixteen times, from 25 to 400 entries; the mask sets every entry above the diagonal to exactly zero, so a position never attends to a later one

The matrix is n × n per head per layer, so 20 tokens give 400 entries where 5 gave 25. The script checks the mask by printing the largest above-diagonal entry, which is 0.000000, not small: the masked scores are minus infinity before the softmax and exp(−∞) is zero.

Question 4. The top token had probability 0.66 at temperature 1.0. Which line is consistent with what temperature does to the same logits?

• T = 0.5 gives 0.33, because halving the temperature halves the probability
• T = 0.5 gives 0.99 and T = 1.5 gives 0.16, because dividing every logit by T stretches or compresses the gaps between them before the softmax
• T = 0.5 gives 0.66, because temperature only affects sampling, not probabilities
• T = 1.5 gives 0.99, because higher temperature makes the model more confident

Show the answer and why

Answer: T = 0.5 gives 0.99 and T = 1.5 gives 0.16, because dividing every logit by T stretches or compresses the gaps between them before the softmax

p_i = exp(z_i / T) / Σ exp(z_j / T). A T below one widens the gaps and the top token takes almost everything; a T above one narrows them and the distribution flattens, which the entropy column shows rising from 2.29 to 7.10 nats in the dry run.

Question 5. On Track M, `mlx_lm.generate --model ~/llm-course/models/qwen3-1.7b-base --prompt "What is the capital of Brazil?"` replies like an assistant instead of continuing the text. Which command is the fix?

• Add --chat-template-config '{"enable_thinking": false}'
• Add --ignore-chat-template, because the base repository ships a chat template and mlx_lm.generate applies it by default
• Add --max-tokens 300
• Point --model at the instruct directory instead

Show the answer and why

Answer: Add --ignore-chat-template, because the base repository ships a chat template and mlx_lm.generate applies it by default

mlx_lm.generate wraps the prompt in the tokeniser's chat template whenever one exists, and the base repository's tokenizer_config.json carries one. --ignore-chat-template sends the seven raw tokens, which is what task 7 did with Transformers by calling the tokeniser directly.

Question 6. The base model answered your question with eleven more questions, and the instruct model answered it directly. Which explanations are consistent with what you saw? Select all that apply.

• The base model is continuing text, which is what pretraining optimised, and greedy decoding makes the repetition visible
• The instruct model received the question inside a chat template whose eight control tokens it was post-trained on
• The base model is a smaller architecture with fewer layers
• Post-training changed the weight values so that an answer, or a thinking block, is the likely continuation of that token pattern

Show the answer and why

Answer: The base model is continuing text, which is what pretraining optimised, and greedy decoding makes the repetition visible, The instruct model received the question inside a chat template whose eight control tokens it was post-trained on, Post-training changed the weight values so that an answer, or a thinking block, is the likely continuation of that token pattern

You listed both checkpoints: 28 layers, the same shapes, the same 1,409,410,048 non-embedding parameters, and the same tokeniser ids. What differs is the values of the weights and the string the model was given, and the model card describes the base as being at the pretraining stage with no post-training.

Check my answersClear