Embeddings: Meaning as Geometry

By the end of this lesson you will be able to say what happens between a token id and the first matrix multiplication and count the parameters it costs from two fields of config.json; watch one token’s vector change layer by layer as its neighbours change; compute cosine similarity by hand and say exactly what it measures; read a config.json and a repository’s pooling files to tell an embedding model from a generative one; put a number on what “similar” does and does not imply, from your own run; and work out what an index of a million vectors costs before you build one. Retrieval in Part 10, agent memory in Part 24 and the evaluation of both rest on this lesson.

Three of the snippets below load a model, and need the environment step and the download from tasks 1 and 2 of this part’s lab; run them after the lab, or now if it is done, from a file saved in ~/llm-course (Track S: /workspace/course) with that environment active. Track S: pass the model directories explicitly, under /workspace/course/models/ (for example python static-vs-contextual.py /workspace/course/models/qwen3-1.7b), because the scripts’ defaults expand ~ to the container’s own home directory; the same substitution applies to the download path below. The versions this page was written against are transformers 5.16.1 · verified 2026-09-08 and Hugging Face CLI 1.30.0 · verified 2026-09-08. The embedding model is one extra download, about 1.2 GB, Apache-2.0 and not gated:

RunnableAll tracks

one extra download for this lesson: the embedding model

hf download Qwen/Qwen3-Embedding-0.6B --local-dir ~/llm-course/models/qwen3-embedding-0.6b

The outputs shown are the author’s dry run on Qwen3-0.6B (Apache-2.0), the lab’s reduced path, on a CPU in BF16 with transformers 5.16.1 on 2026-09-12. On Qwen3-1.7B (Apache-2.0) every cosine and every vector length will differ, the width printed is 2048 rather than 1024, and cross-model.py skips its two cross-model lines because the widths no longer match; the entry count (29), the depths and the token id (6073) are the same.

A token id is a row number

The tokens lesson ended with a list of integers. A model cannot multiply an integer id by anything meaningful, because ids are arbitrary labels: token 4,102 is not twice token 2,051, and “ bank” being id 6073 says nothing about banks.

The first thing a model does, therefore, is a lookup. In the Qwen3 code at the pinned Transformers tag the embedding layer is constructed as nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx): one row per vocabulary entry, one column per dimension of the model’s internal width, and the third argument is the padding token’s id. The token id selects a row. That row is a vector of learned numbers, and it is the model’s entire representation of the token before any context is taken into account. The first two fields are in every config.json you will read, and one more, tie_word_embeddings, decides whether the matrix at the other end of the stack is a second copy:

embedding parameters  = vocab_size × hidden_size
output projection     = a second vocab_size × hidden_size matrix, unless tie_word_embeddings is true,
                        in which case the same matrix is used in both places

The output projection is the matrix that turns the final vector back into one logit per vocabulary entry, the distribution the next-token lesson described. modeling_qwen3.py declares the tie as _tied_weights_keys = {"lm_head.weight": "model.embed_tokens.weight"}, so when the flag is true the head is not a second matrix but a second name for the first. Run the arithmetic for four models; the three fields per model were read from each config.json on the Hub, the card figures from each model card, and the rest is computed:

RunnableAll tracks

embedding-matrix-count.py

"""Count the embedding matrix from config.json fields and reconcile it with the model card."""
# The three fields below were read from each model's config.json on the Hub (2026-09-12);
# the card figures are what the model cards state. Everything else is arithmetic.
models = {
    #  name                    hidden_size  vocab_size  tie_word_embeddings  card total  card non-embedding
    "Qwen3-8B":              (4096,        151936,     False,               8.2e9,      6.95e9),
    "Qwen3-1.7B":            (2048,        151936,     True,                1.7e9,      1.4e9),
    "Qwen3-0.6B":            (1024,        151936,     True,                0.6e9,      0.44e9),
    "Qwen3-Embedding-0.6B":  (1024,        151669,     True,                0.6e9,      None),
}
print(f"{'model':<22s} {'rows x cols':>16s} {'one matrix':>13s} {'tied':>5s} {'in-and-out':>13s} {'card gap':>10s}")
for name, (hidden, vocab, tied, total, non_emb) in models.items():
    one = vocab * hidden                      # parameters in the input embedding matrix
    both = one if tied else 2 * one           # plus the output projection, unless it is the same matrix
    gap = f"{(total - non_emb) / 1e9:.2f}B" if non_emb else "n/a"
    print(f"{name:<22s} {vocab:>7,} x {hidden:<6,} {one:>13,} {str(tied):>5s} {both:>13,} {gap:>10s}")
print("\nbytes for one matrix, Qwen3-8B, by element size:")
for label, nbytes in (("FP32", 4), ("BF16", 2)):
    print(f"  {label:<6s} {151936 * 4096 * nbytes / 1e9:6.2f} GB")

Output — what you should see

model                       rows x cols    one matrix  tied    in-and-out   card gap
Qwen3-8B               151,936 x 4,096    622,329,856 False 1,244,659,712      1.25B
Qwen3-1.7B             151,936 x 2,048    311,164,928  True   311,164,928      0.30B
Qwen3-0.6B             151,936 x 1,024    155,582,464  True   155,582,464      0.16B
Qwen3-Embedding-0.6B   151,669 x 1,024    155,309,056  True   155,309,056        n/a

bytes for one matrix, Qwen3-8B, by element size:
  FP32     2.49 GB
  BF16     1.24 GB

Read the in-and-out column against the card gap column. Qwen3-8B (Apache-2.0) unties its embeddings, so two matrices of 622 million sit either side of the stack, and the card’s gap between total and non-embedding parameters is about twice one matrix. Qwen3-1.7B and Qwen3-0.6B tie theirs, and their gaps are about one matrix each. The agreement is the point: two fields in a configuration file and one subtraction on a model card give the same number, and if they did not, one of them would be wrong. The embedding model’s row count is different because its config.json declares the tokeniser’s actual size, 151,669, where the generative checkpoints pad the matrix to 151,936, the difference the tokens lesson explained.

The lookup table alone is over six hundred million parameters on Qwen3-8B, and in the BF16 the checkpoint ships in, over a gigabyte of the file, the bytes-per-element arithmetic from Part 1. A checkpoint can also carry the tied matrix twice: Qwen3-1.7B’s second safetensors shard stores lm_head.weight even though the flag says it is the same matrix, which is why the lab’s list-tensors.py prints whether the head is stored in the file. The lab has you confirm the same numbers a third way, from the tensor shapes in the file.

Static rows and contextual vectors

The embedding row for a token is the same every time that token appears. In “She sat on the bank of the river” and “She paid the cheque into the bank”, the row selected for bank is identical. This is the static or input embedding, a token embedding, and on its own it cannot distinguish the two senses.

What happens next is the point of the whole architecture. Each layer reads the vectors at all the positions, lets each one gather information from the others through attention, and writes an updated vector back at every position. After the first layer the vector at the position of bank is no longer the dictionary row; it has been modified by the presence of river or cheque nearby. That is a contextual embedding, one vector per position per layer, and the attention lesson explains the mechanism that does the gathering.

Transformers exposes every stage. The documented hidden_states output is “a tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size)”. Loading a checkpoint with AutoModel rather than AutoModelForCausalLM gives you the stack without its output head, which is all this experiment needs. The script follows one token through three sentences, two with the river sense and one with the money sense:

RunnableAll tracks

static-vs-contextual.py

"""One token, three sentences: the static embedding row, then the vector after each layer."""
import sys
from pathlib import Path

import torch
from transformers import AutoModel, AutoTokenizer

model_dir = Path(sys.argv[1] if len(sys.argv) > 1 else "~/llm-course/models/qwen3-1.7b").expanduser()
tokeniser = AutoTokenizer.from_pretrained(model_dir)
model = AutoModel.from_pretrained(model_dir, dtype=torch.bfloat16)   # the stack without its output head
model.eval()

sentences = {
    "river":  "She sat on the bank of the river.",
    "money":  "She paid the cheque into the bank.",
    "river2": "The river burst its bank after the storm.",
}
bank_id = tokeniser.encode(" bank")[0]          # one token, with its leading space

vectors = {}                                     # name -> the 'bank' vector at every depth
for name, text in sentences.items():
    enc = tokeniser(text, return_tensors="pt")
    pos = enc["input_ids"][0].tolist().index(bank_id)
    with torch.no_grad():
        out = model(**enc, output_hidden_states=True)
    vectors[name] = [h[0, pos].float() for h in out.hidden_states]

layers, width = model.config.num_hidden_layers, model.config.hidden_size
print(f"token ' bank' is id {bank_id}; hidden_states has {len(out.hidden_states)} entries "
      f"(embedding output + {layers} layers), each {width} wide")
row = model.embed_tokens.weight[bank_id].float()
print(f"entry 0 equals row {bank_id} of embed_tokens.weight: {torch.equal(vectors['river'][0], row)}")
print(f"last entry is last_hidden_state: {out.hidden_states[-1] is out.last_hidden_state}")

cos = torch.nn.functional.cosine_similarity
print(f"\n{'depth':>5s}  {'river vs money':>14s}  {'river vs river2':>15s}  {'|river|':>8s}   (cosine between 'bank' vectors; length of one)")
for depth in sorted({0, 1, 2, 4, layers // 4, layers // 2, 3 * layers // 4, layers}):
    a, b, c = (vectors[k][depth] for k in ("river", "money", "river2"))
    print(f"{depth:5d}  {cos(a, b, dim=0):14.4f}  {cos(a, c, dim=0):15.4f}  {a.norm():8.2f}")

RunnableAll tracks

run static-vs-contextual.py

# default: Qwen3-1.7B, the lab's download
python static-vs-contextual.py
# the reduced path, the run shown below
python static-vs-contextual.py ~/llm-course/models/qwen3-0.6b

Output — what you should see

Loading weights: 100%|██████████| 310/310 [00:00<00:00, xxxx.xxit/s]
[transformers] Qwen3Model LOAD REPORT from: /home/you/llm-course/models/qwen3-0.6b
Key            | Status     |  |
---------------+------------+--+-
lm_head.weight | UNEXPECTED |  |

Notes:
- UNEXPECTED:  can be ignored when loading from different task/architecture; not ok if you expect identical arch.
token ' bank' is id 6073; hidden_states has 29 entries (embedding output + 28 layers), each 1024 wide
entry 0 equals row 6073 of embed_tokens.weight: True
last entry is last_hidden_state: True

depth  river vs money  river vs river2   |river|   (cosine between 'bank' vectors; length of one)
    0          1.0000           1.0000      0.90
    1          0.9858           0.9155      8.55
    2          0.9752           0.9180     11.03
    4          0.8312           0.8568     14.21
    7          0.6706           0.7840     22.00
   14          0.7072           0.7949     54.21
   21          0.7628           0.8356    192.22
   28          0.6133           0.7972    106.64

The Loading weights line is Transformers’ own progress bar, and the load report is the library noticing that the file carries an lm_head.weight the headless Qwen3Model has no slot for; its own note says it can be ignored, and on Qwen3-1.7B it appears for the same reason. Three things to read off the table:

Depth	What the numbers say
0	Both cosines are exactly 1: the three occurrences are the same row, and the True above confirms entry 0 of hidden_states is that row of embed_tokens.weight, unchanged.
1 to 4	The vectors diverge within a few layers, and at depths 1 and 2 the two river sentences are further apart than river and money are; whatever the early layers write in first, it is not the sense of the word.
28	By the last layer the river-versus-money pair sits well below the river-versus-river pair. The same token, given different neighbours, has become two different vectors, and the one whose neighbours agree stays closer.

Use the depth-0 row, or an early layer, as the embedding of a text and the two senses of bank are indistinguishable (cosine 1.0000 at depth 0); that is the failure an embedding model’s “use the last hidden state” recipe exists to prevent.

The length column is the reason the next section divides by length: the same token’s vector goes from under one at depth 0 to over a hundred at the end, and the last entry is the final normalisation’s output (at the pinned version it is the same tensor as last_hidden_state, which the script’s last entry is last_hidden_state: True line checks). A comparison that ignored length would mostly be comparing depth.

From an id to a vector, and what the vector is used for

1. 1Token idAn integer chosen by the tokeniser.
2. 2Embedding lookupRow of the vocab_size × hidden_size matrix. The same row every time.
3. 3Transformer layersEach position’s vector absorbs information from the others. One vector per position, per layer.
4. 4Final vectorsOne contextual vector per position, hidden_size wide.
5. 5Output headProject to the vocabulary for generation, or pool into one vector for an embedding.

The same stack serves both purposes. A generative model ends by projecting the final vector to one logit per vocabulary entry; an embedding model ends by pooling the final vectors into one vector for the whole text. The difference is the last step and the training, not the middle.

Cosine similarity, exactly

Once text is a vector, “similar” becomes a measurement. The standard one is cosine similarity: the dot product of two vectors divided by the product of their lengths, which is the cosine of the angle between them.

cos(u, v) = (u · v) / (|u| × |v|)
u · v     = u₁v₁ + u₂v₂ + … + uₙvₙ          (the dot product: multiply element-wise, add up)
|u|       = sqrt(u₁² + u₂² + … + uₙ²)        (the length, or L2 norm)
range     = 1 (same direction) … 0 (perpendicular) … −1 (opposite direction)

Three-dimensional vectors are enough to see every case, and the arithmetic is small enough to check on paper:

RunnableAll tracks

cosine-by-hand.py

"""Cosine similarity on three-dimensional vectors, every intermediate number printed."""
import math

a = [2.0, 1.0, 0.0]
b = [4.0, 2.0, 0.0]      # the same direction as a, twice as long
c = [0.0, 1.0, 2.0]
d = [-2.0, -1.0, 0.0]    # a, pointing the other way

def dot(u, v):
    return sum(ui * vi for ui, vi in zip(u, v))

def norm(u):
    return math.sqrt(dot(u, u))

def cosine(u, v):
    return dot(u, v) / (norm(u) * norm(v))

print(f"{'pair':<5s} {'dot':>6s} {'|u|':>6s} {'|v|':>6s} {'cos':>8s} {'euclid':>7s}")
for name, (u, v) in {"a,b": (a, b), "a,c": (a, c), "a,d": (a, d), "a,a": (a, a)}.items():
    euclid = math.sqrt(sum((ui - vi) ** 2 for ui, vi in zip(u, v)))
    print(f"{name:<5s} {dot(u, v):6.2f} {norm(u):6.3f} {norm(v):6.3f} {cosine(u, v):8.4f} {euclid:7.4f}")

unit = lambda u: [ui / norm(u) for ui in u]        # divide by the length first ...
print(f"\nunit(a) . unit(c) = {dot(unit(a), unit(c)):.4f}   ... and the cosine is just a dot product")

Output — what you should see

pair     dot    |u|    |v|      cos  euclid
a,b    10.00  2.236  4.472   1.0000  2.2361
a,c     1.00  2.236  2.236   0.2000  2.8284
a,d    -5.00  2.236  2.236  -1.0000  4.4721
a,a     5.00  2.236  2.236   1.0000  0.0000

unit(a) . unit(c) = 0.2000   ... and the cosine is just a dot product

The a,b row is the one to remember. The two vectors point the same way, so their cosine is 1, while their Euclidean distance is not zero because one is twice as long. Dividing by the lengths makes cosine a comparison of direction and nothing else, which is what lets a short query be compared with a long passage, and what made the depth-0 and depth-28 vectors above comparable at all. The last line is why retrieval systems are fast: normalise every vector to unit length once, when it is stored, and the cosine between a query and a million stored vectors is one matrix multiplication. The Qwen3-Embedding-0.6B usage example ends with exactly that normalisation.

Measure	Use it when	Do not use it when
Cosine	Comparing text embeddings; the model was trained so that direction carries meaning	You need magnitude: cosine cannot tell a from b above
Dot product	Every stored vector is unit length, so it equals cosine and costs one multiply-add per dimension	Vectors were not normalised: longer vectors then win regardless of direction
Euclidean distance	Magnitude is meaningful, as in many non-text feature spaces	Comparing the outputs of a model that normalises, where it is a monotone function of cosine and buys nothing

Embedding models and generative models

The stack in the diagram is the same for both. Put the two config.json files side by side and the architecture is identical down to the head counts; what differs is the vocabulary padding, the position limit and the end-of-text id:

config.json field	Qwen3-0.6B (generative)	Qwen3-Embedding-0.6B (embedding)
architectures	Qwen3ForCausalLM	Qwen3ForCausalLM
hidden_size / num_hidden_layers	1024 / 28	1024 / 28
num_attention_heads / num_key_value_heads / head_dim	16 / 8 / 128	16 / 8 / 128
intermediate_size	3072	3072
vocab_size	151936	151669
max_position_embeddings	40960	32768
tie_word_embeddings	true	true
eos_token_id	151645	151643

The differences that matter are not in config.json. They are in what sits after the last layer and in what the training asked for.

A generative model projects the final vector at the last position through the output matrix, one logit per vocabulary entry, and is trained to make the true next token likely.

An embedding model pools the final vectors into one vector for the whole input and is trained so that texts which should match land close together and texts which should not land apart. The Qwen3-Embedding-0.6B repository records its recipe in files: modules.json lists a Transformer step, a Pooling step and a Normalize step, in that order; 1_Pooling/config.json sets pooling_mode_lasttoken to true and every other pooling mode to false; and the safetensors header lists 310 tensors, embed_tokens.weight of shape [151669, 1024], the 28 blocks and the final norm.weight, with no lm_head.weight at all. The output head is gone because nothing is generated. Its tokeniser appends <|endoftext|> (id 151643) to every input, so “the last token” is that marker, the one position whose vector has attended to the whole text.

The card’s spec table says 0.6B parameters, 28 layers, a sequence length of 32K and an embedding dimension of up to 1024, with MRL support and instruction awareness. Instruction awareness means the query carries a short task description; the repository’s own prompt file gives it verbatim as Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: on the query side and an empty prefix on the document side, and the card reports that “not using an instruct on the query side can lead to a drop in retrieval performance by approximately 1% to 5%”, a claim Part 10 has you test on your own documents. MRL support means the vector can be cut short: the card describes user-defined output dimensions from 32 to 1024. The script below does what the card’s usage example does, then does three things the card does not, so that you can see what each choice changes:

RunnableAll tracks

embed-and-compare.py

"""Embed a query and four passages with Qwen3-Embedding-0.6B and compare them four ways."""
import sys
from pathlib import Path

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_dir = Path(sys.argv[1] if len(sys.argv) > 1 else "~/llm-course/models/qwen3-embedding-0.6b").expanduser()
tokeniser = AutoTokenizer.from_pretrained(model_dir, padding_side="left")   # as the card's example does
model = AutoModel.from_pretrained(model_dir, dtype=torch.bfloat16)
model.eval()

task = "Given a web search query, retrieve relevant passages that answer the query"
texts = [
    f"Instruct: {task}\nQuery:how do I reset a forgotten password",                          # 0: the query
    "To reset a forgotten password, open Settings, choose Account and click Reset password.",  # 1: the answer
    "Passwords must contain at least twelve characters and one digit.",                        # 2: same topic, no answer
    "The service was excellent and the staff were helpful.",                                   # 3
    "The service was terrible and the staff were unhelpful.",                                  # 4: the opposite of 3
]

def embed(texts, dims=None, pooling="last"):
    enc = tokeniser(texts, padding=True, truncation=True, max_length=8192, return_tensors="pt")
    with torch.no_grad():
        h = model(**enc).last_hidden_state.float()           # (batch, seq, 1024): one vector per position
    if pooling == "last":
        v = h[:, -1]                                          # left padding puts every last token at -1
    else:
        m = enc["attention_mask"].unsqueeze(-1).float()
        v = (h * m).sum(1) / m.sum(1)                         # mean over the real tokens
    if dims:
        v = v[:, :dims]                                       # MRL: keep the first dims, then renormalise
    return F.normalize(v, p=2, dim=1)                         # unit length, so cosine == dot product

def show(title, v):
    s = v @ v.T
    print(f"\n{title}")
    print("     " + "".join(f"{j:>7d}" for j in range(len(texts))))
    for i in range(len(texts)):
        print(f"  {i:>2d} " + "".join(f"{s[i, j]:7.3f}" for j in range(len(texts))))

show("last-token pooling, 1024 dims (the card's recipe)", embed(texts))
show("last-token pooling, first 256 dims", embed(texts, dims=256))
show("last-token pooling, first 64 dims", embed(texts, dims=64))
show("mean pooling, 1024 dims (not how this model was trained)", embed(texts, pooling="mean"))

RunnableAll tracks

run embed-and-compare.py

# the default is ~/llm-course/models/qwen3-embedding-0.6b, the download above
python embed-and-compare.py

Output — what you should see

Loading weights: 100%|██████████| 310/310 [00:00<00:00, xxxx.xxit/s]

last-token pooling, 1024 dims (the card's recipe)
           0      1      2      3      4
   0   1.000  0.823  0.397  0.106  0.181
   1   0.823  1.000  0.495  0.206  0.268
   2   0.397  0.495  1.000  0.295  0.305
   3   0.106  0.206  0.295  1.000  0.790
   4   0.181  0.268  0.305  0.790  1.000

last-token pooling, first 256 dims
           0      1      2      3      4
   0   1.000  0.826  0.420  0.148  0.232
   1   0.826  1.000  0.483  0.252  0.321
   2   0.420  0.483  1.000  0.325  0.353
   3   0.148  0.252  0.325  1.000  0.792
   4   0.232  0.321  0.353  0.792  1.000

last-token pooling, first 64 dims
           0      1      2      3      4
   0   1.000  0.891  0.484  0.123  0.274
   1   0.891  1.000  0.511  0.174  0.258
   2   0.484  0.511  1.000  0.230  0.337
   3   0.123  0.174  0.230  1.000  0.790
   4   0.274  0.258  0.337  0.790  1.000

mean pooling, 1024 dims (not how this model was trained)
           0      1      2      3      4
   0   1.000  0.664  0.623  0.588  0.591
   1   0.664  1.000  0.654  0.447  0.475
   2   0.623  0.654  1.000  0.445  0.467
   3   0.588  0.447  0.445  1.000  0.916
   4   0.591  0.475  0.467  0.916  1.000

The first matrix is the model doing its job. Row 0 is the query, and its nearest passage by a wide margin is the one that answers it; the passage on the same topic that does not answer it is next; the two sentences about service are far away. Read the same matrix for the two service sentences and the number is uncomfortable, which the next section is about.

Which vectors, and pooled how

Two choices turn a stack of contextual vectors into one embedding, and both are recorded on a model card or in the repository rather than being universal.

The first is which layer. The hidden_states tuple has one entry per layer plus the embedding output; an embedding model is trained end to end with one choice in mind, and for this model it is the final normalised output, last_hidden_state.

The second is how to pool. Last-token pooling is what this model was trained with; averaging the vectors across all positions is the other common choice and is usual for encoder-style models. The two are not interchangeable, and the fourth matrix above shows what “not interchangeable” looks like:

Pair	Last-token, 1024 dims	Mean-pooled, 1024 dims
query vs the answer (0, 1)	0.823	0.664
query vs same topic, no answer (0, 2)	0.397	0.623
query vs an unrelated sentence (0, 3)	0.106	0.588
the two opposite service sentences (3, 4)	0.790	0.916

With the wrong pooling the query’s scores collapse into a narrow band and the answer is within eight hundredths of an unrelated sentence. That is the symptom to recognise: when everything you compare scores about the same, check the pooling and the instruction prefix against the card before you blame the model.

The same reasoning says why pooling a generative model is a poor substitute for a model trained to embed, and you can measure it rather than take it on trust, since Qwen3-0.6B and Qwen3-Embedding-0.6B share a width of 1024:

RunnableAll tracks

cross-model.py

"""Two models, the same width, different spaces: a cosine across them is a number that means nothing."""
import sys
from pathlib import Path

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

embed_dir = Path(sys.argv[1] if len(sys.argv) > 1 else "~/llm-course/models/qwen3-embedding-0.6b").expanduser()
gen_dir = Path(sys.argv[2] if len(sys.argv) > 2 else "~/llm-course/models/qwen3-1.7b").expanduser()

texts = ["To reset a forgotten password, open Settings, choose Account and click Reset password.",
         "The service was terrible and the staff were unhelpful."]

def last_token_vectors(model_dir):
    tokeniser = AutoTokenizer.from_pretrained(model_dir, padding_side="left")
    model = AutoModel.from_pretrained(model_dir, dtype=torch.bfloat16)
    model.eval()
    enc = tokeniser(texts, padding=True, return_tensors="pt")
    with torch.no_grad():
        h = model(**enc).last_hidden_state.float()
    return F.normalize(h[:, -1], p=2, dim=1)          # (2, hidden_size), unit length

e, g = last_token_vectors(embed_dir), last_token_vectors(gen_dir)
print(f"embedding model width {e.shape[1]}, generative model width {g.shape[1]}")
if e.shape[1] == g.shape[1]:
    print(f"same text, embedding model vs generative model: cos = {float(e[0] @ g[0]):.4f}")
    print(f"different texts across the two models:          cos = {float(e[0] @ g[1]):.4f}")
print(f"within the embedding model, texts 0 and 1:      cos = {float(e[0] @ e[1]):.4f}")
print(f"within the generative model, texts 0 and 1:     cos = {float(g[0] @ g[1]):.4f}")

RunnableAll tracks

run cross-model.py

# default: the embedding model against Qwen3-1.7B; the widths differ, so the two cross-model lines are skipped
python cross-model.py
# the 0.6B pair, the run shown below
python cross-model.py \
  ~/llm-course/models/qwen3-embedding-0.6b \
  ~/llm-course/models/qwen3-0.6b

Output — what you should see

Loading weights: 100%|██████████| 310/310 [00:00<00:00, xxxx.xxit/s]
Loading weights: 100%|██████████| 310/310 [00:00<00:00, xxxx.xxit/s]
[transformers] Qwen3Model LOAD REPORT from: /home/you/llm-course/models/qwen3-0.6b
Key            | Status     |  |
---------------+------------+--+-
lm_head.weight | UNEXPECTED |  |

Notes:
- UNEXPECTED:  can be ignored when loading from different task/architecture; not ok if you expect identical arch.
embedding model width 1024, generative model width 1024
same text, embedding model vs generative model: cos = 0.1542
different texts across the two models:          cos = 0.0128
within the embedding model, texts 0 and 1:      cos = 0.2692
within the generative model, texts 0 and 1:     cos = 0.7589

On Qwen3-1.7B the widths differ, 1024 against 2048, and the script skips the cross-model lines because a dot product between them is not even defined; with the 0.6B pair it is defined and tells you nothing, the same sentence scoring close to zero against itself. The last line is the more instructive one: in the generative model two unrelated sentences score high against each other. Its last position was trained to predict what follows a full stop, not to place the sentence’s meaning anywhere in particular, and in this run its last-token vectors point roughly the same way whatever the sentence says. Nothing in that objective asks texts with the same meaning to land together; the embedding model’s objective asks for nothing else. If you must pool a generative model, measure it against a trained embedding model on your own data before relying on it.

What “similar” does not imply

Cosine similarity measures proximity in a space that was shaped by a training objective. It does not measure truth, relevance to your question, or agreement, and the first matrix above has a number for each of the four ways this goes wrong.

• Opposites are often close. “The service was excellent and the staff were helpful” and “the service was terrible and the staff were unhelpful” share topic, grammar and register and differ in two words, and they score higher against each other than the query scores against the passage on its own topic. Sentiment is not distance, and a retrieval system asked for complaints will happily return praise.
• Similarity is not relevance. The passage about password rules is the query’s second-nearest neighbour. It is about passwords and it answers nothing. In a corpus with many such passages the answer is one of a crowd, which is why Part 10 puts a reranker, a model that reads query and passage together, after the vector search.
• The space is a property of the model. The cross-model run gave a cosine near zero for the same sentence embedded by two models of the same width. Vectors from two embedding models are not comparable even when their dimensions match; re-embed everything when you change models, and record which model and which dimension produced a stored index.
• Scores are relative, not absolute. The gap between the answer and the same-topic passage is this corpus’s gap, with five texts. No threshold means “relevant” across corpora; thresholds are calibrated per corpus and per task on held-out data, the method from Part 1, and re-calibrated when the model, the dimension or the instruction prefix changes.

✕Common mistake

The most expensive version of this mistake is building a retrieval system, measuring nothing, and concluding from a handful of good-looking results that it works. The symptom is a demo that answers the questions its builder tried and a first user who finds the praise-for-complaints failure within an hour. Part 10’s project scores every question of a held-out set three ways before anyone is shown the system.

Dimensionality and what it costs

Dimensions are not free, and the cost is arithmetic you can do before building anything:

index_bytes            = chunks × dims × bytes_per_element
multiply_adds_per_query = chunks × dims                 (a brute-force scan; one dot product per chunk)
scan_time_per_query    ≈ index_bytes / memory_bandwidth (the scan reads the whole index once)

Chunks	Dims	Element	Index size	Multiply-adds per query
1,000,000	1024	FP32 (4 bytes)	4.096 GB	1.024 billion
1,000,000	1024	FP16 (2 bytes)	2.048 GB	1.024 billion
1,000,000	256	FP16	0.512 GB	0.256 billion
1,000,000	64	FP16	0.128 GB	0.064 billion
100,000	1024	FP16	0.205 GB	0.102 billion

These are computed from the formula, not measured. The third line is the reason the scan time is written in terms of bandwidth: a brute-force search streams the whole index past the processor once per query, the same bytes-moved arithmetic that Part 1 used for a matrix multiplication and that Part 5 will use for decode speed, and an index that fits in memory is searched at memory speed while one that does not is searched at disk speed. Vector databases can add index structures that avoid the full scan; Part 10’s store is a single file, and the arithmetic still decides whether it fits in memory.

More dimensions give the model more room to separate things, up to a point that depends on the model and the data, and MRL training makes the trade adjustable after the fact. The second and third matrices from the run are the same five texts at 256 and 64 dimensions, normalised again after truncation, which the card’s dimension range allows:

Pair	1024 dims	256 dims	64 dims
query vs the answer (0, 1)	0.823	0.826	0.891
query vs same topic, no answer (0, 2)	0.397	0.420	0.484
query vs an unrelated sentence (0, 3)	0.106	0.148	0.123
the two opposite service sentences (3, 4)	0.790	0.792	0.790

At a quarter of the storage the ranking is unchanged and the scores move by a few hundredths; at a sixteenth the scores drift more but the order still holds, on these five texts. That last clause is the decision rule: truncation is a storage-and-speed decision whose cost in ranking quality is measured on your corpus with a held-out question set, never assumed from a five-text demonstration or from a card. A common shape is a first pass over short vectors and a second pass, over the full vectors or through a reranker, for the top few hundred candidates. If you truncate, renormalise; a truncated vector is no longer unit length and its dot products are no longer cosines.

The same width runs through a generative model. hidden_size is the width of every vector in the stack, and it sets the size of the embedding matrix, the attention projections and the feed-forward blocks; it is one of the two or three numbers that decide how large a model is, which is where the last lesson in this part starts.

Where embeddings come back

Part	What it does with embeddings	Page
10	Serves Qwen3-Embedding-0.6B from llama-server with --embedding, chunks documents, stores the vectors in a vector table with cosine distance, ranks by similarity and hands the top passages to a generative model behind a reranker	Retrieval-augmented generation
10	Scores the retrieval system against a held-out question set, refusal, contained fact and cited source, so a change to chunking, dimension or reranking becomes a number	Project: private document QA
16	Judges answers against a reference with a rubric and a judge model; similarity to a reference answer is where the temptation to score by embedding distance arises, and this lesson is why that number is not a score	LLM-as-judge
24	Turns the Part 10 index into an agent’s long-term memory: the agent asks and gets back the passages that answer, instead of carrying the corpus in its context	Context engineering, memory and compaction
26	Lets the agent decide what to retrieve and when, with the same instruction-prefixed query embeddings	Agentic retrieval and research agents

A retrieval error you can explain geometrically

Imagine searching a maintenance manual for “disable remote access”. A passage describing how to enable remote access shares much of the same vocabulary and topic, so it can be close in embedding space. Similarity is useful for selecting candidates, but the sign of an instruction can decide whether a candidate actually answers the question.

Use three stages of evidence. First, check whether the relevant passage appears anywhere in the retrieved candidates. Second, check whether reranking places it among the passages sent to the generator. Third, check whether the answer preserves the passage’s conditions and negations. A failure at each stage requires a different repair; changing the generator cannot retrieve a missing page.

Also keep vector spaces separate. Two embedding models may produce vectors with identical dimensions but unrelated coordinate systems. Matching dimensions is only a storage check. Record the model revision, normalisation and query/document preprocessing with the index, and rebuild document vectors when that contract changes. Otherwise the database may accept the query while its distances cease to have the meaning you intended.

Recap

A token id selects a row of the embedding matrix, vocab_size by hidden_size, and tie_word_embeddings says whether the output projection is that same matrix or a second one; the two settings explain the gap between total and non-embedding parameters on every Qwen3 card. The row is static; the layers rewrite it with its context, and by the last layer the same token in two senses has become two vectors. Cosine similarity divides the dot product by the lengths and measures direction alone, which is why it survives the hundredfold growth in vector length through the stack and why unit-normalised stores search with one matrix multiplication. An embedding model is the same stack with the output head removed, a pooling step the repository records, and an objective that asks matching texts to land together; pool it the way its card says or the scores collapse into a band. Each thing a score does not imply has a number in this lesson’s run. Dimensions cost bytes and multiply-adds per query, and truncation trades them against ranking quality; Parts 10, 16, 24 and 26 pick the thread up.

Check your understanding

Question 1. Qwen3-1.7B declares hidden_size 2048, vocab_size 151936 and tie_word_embeddings true. Its card gives 1.7B total and 1.4B non-embedding parameters. Which statement is consistent with all of that?

• The embedding matrix holds about 0.3 billion parameters and is reused as the output projection
• The embedding matrix holds about 0.3 billion parameters and a second matrix of the same size is used at the output
• The embedding matrix holds about 1.4 billion parameters
• The vocabulary size does not affect the parameter count

Show the answer and why

Answer: The embedding matrix holds about 0.3 billion parameters and is reused as the output projection

151,936 × 2,048 is 311,164,928, about 0.3 billion, which is the gap between total and non-embedding on the card. tie_word_embeddings true means one matrix does both jobs; Qwen3-8B sets it false and its gap is about twice its matrix. A checkpoint may still store lm_head.weight as a copy, which changes the file, not the count.

Question 2. In static-vs-contextual.py the cosine between the two "bank" vectors is exactly 1.0000 at depth 0 and 0.6133 at depth 28. What produced the change?

• The tokeniser assigned different ids to the two occurrences
• The embedding row was updated during the forward pass
• Each layer let the position read from its neighbours, so the vector at "bank" absorbed "river" in one sentence and "cheque" in the other
• Cosine similarity is unreliable on long vectors

Show the answer and why

Answer: Each layer let the position read from its neighbours, so the vector at "bank" absorbed "river" in one sentence and "cheque" in the other

The id is 6073 in both sentences and entry 0 of hidden_states is the unchanged row of embed_tokens.weight, as the script printed. Nothing in the matrix changes during inference; the layers write new vectors at each position from the old ones and their neighbours. That is the difference between a token embedding and a contextual one.

Question 3. Two sentences have a cosine similarity of 0.79 in Qwen3-Embedding-0.6B. Which conclusions are unsafe? Select all that apply.

• They are about a similar topic in this model’s space
• They agree with each other
• One answers the question posed by the other
• A vector from a different embedding model with the same score means the same thing

Show the answer and why

Answer: They agree with each other, One answers the question posed by the other, A vector from a different embedding model with the same score means the same thing

That is the score the run gave "the service was excellent" against "the service was terrible". Proximity is topic-and-form similarity in one model’s space; opposites score highly because they share everything except the crucial words, relevance is a separate measurement, and the cross-model run gave a cosine near zero for the same sentence in two models.

Question 4. Each line embeds texts with Qwen3-Embedding-0.6B for a retrieval index. Which one is the bug?

• tokeniser = AutoTokenizer.from_pretrained(model_dir, padding_side="left"); v = h[:, -1]
• m = mask.unsqueeze(-1).float(); v = (h * m).sum(1) / m.sum(1)
• v = F.normalize(v, p=2, dim=1); scores = q @ v.T
• query = f"Instruct: {task}\nQuery:{question}"; documents are embedded with no prefix

Show the answer and why

Answer: m = mask.unsqueeze(-1).float(); v = (h * m).sum(1) / m.sum(1)

That is mean pooling, and the repository’s 1_Pooling/config.json sets pooling_mode_lasttoken and nothing else. The run showed what it does: the query’s four scores bunch between 0.59 and 0.66 and the answer barely beats an unrelated sentence. Left padding with the last position, normalise-then-dot, and the instruction on the query side only are all the card’s recipe.

Question 5. An index holds 1,000,000 chunks at 1024 dimensions in FP16. You truncate to 256 dimensions. What is the new index size, and what else must change?

• 1.024 GB, and nothing else; the stored vectors are still unit length
• 0.512 GB, and every truncated vector must be renormalised, then the ranking re-measured on a held-out set
• 0.512 GB, and the model must be re-trained at 256 dimensions
• 2.048 GB, because dimension does not change storage

Show the answer and why

Answer: 0.512 GB, and every truncated vector must be renormalised, then the ranking re-measured on a held-out set

1,000,000 × 256 × 2 bytes is 0.512 GB, a quarter of 2.048 GB. Cutting a unit vector short leaves it shorter than unit length, so its dot products are no longer cosines until it is normalised again; the card’s MRL training is what makes the first 256 dimensions usable without re-training, and whether the ranking survives on your corpus is a measurement, not a property of the card.

Question 6. Qwen3-0.6B and Qwen3-Embedding-0.6B both produce 1024-wide vectors, so a query embedded with one can be searched against an index built with the other.

• True
• False

Show the answer and why

Answer: False

The cross-model run gave a cosine of about 0.15 for the same sentence across the two models and about 0.01 for different sentences: the dot product is defined and meaningless. Each model’s space is its own; an index records which model and which dimension built it, and is rebuilt when either changes.

Check my answersClear