Glossary
Terms are defined the way this course uses them, which is sometimes narrower than common usage and occasionally at odds with it. Where a word is genuinely ambiguous in the wider literature, the entry says so.
Entries the course teaches link to the lesson. “See also” links and the links inside definitions resolve within this page, and the test suite fails if any of them points at a term that is not here. Use the filter below to narrow the list by category or by any word in a term or its definition.
132 terms in 10 categories.129 of them are taught in this course and link to the lesson; the rest are here because a reader of trading material will meet them, and a definition that is honest about what a word does and does not mean is worth having to hand.
A
- ActivationMachine learning
The values a layer outputs for a given input, as opposed to the weights that produced them. Activations are what the KV cache stores during inference and what training keeps in memory between the forward and backward passes; their size scales with batch and sequence length.
- Activation functionalso: ReLU, GELU, SiLUMachine learning
The non-linear function applied after a layer's weighted sum: ReLU, GELU, SiLU and their relatives. Without it, stacked linear layers would collapse into one linear layer and depth would add nothing. Not to be confused with Activation, the values flowing through the network.
- Active parametersModels
The parameters actually used for one token in a mixture-of-experts model, the "A3B" in Qwen3-30B-A3B. They set the bytes read per token and therefore the decode speed; the total parameters set the memory needed. A dense model's two counts are equal.
- Attentionalso: Self-attentionModels
The mechanism that lets each token gather information from every earlier token, weighting them by relevance computed from queries and keys and summing their values. Its cost grows with the square of the sequence length, which is why long contexts are expensive.
B
- Backpropagationalso: Backward passMachine learning
The algorithm that computes the gradient of the loss with respect to every parameter in one backward sweep, by applying the chain rule from the output back through each layer. It is what
loss.backward()does in PyTorch and what MLX's value-and-gradient transform does.- Bandwidth-boundalso: Memory-boundHardware
A workload whose speed is limited by how fast bytes can be fetched rather than by arithmetic. Decoding one token at a time is bandwidth-bound: one multiply-add per weight, but every weight must be read from memory. Contrast Compute-bound.
- Base modelalso: Pretrained model, Foundation modelModels
A pretrained model before any post-training: it continues text rather than answering, and is the right starting point for fine-tuning and the wrong one for chat. Distinguished by the absence of "Instruct" or "it" in the name.
- Batch sizeMachine learning
The number of examples one gradient is computed from before a step is taken. It trades gradient noise against cost per step, and because the batch is processed together it is also a memory decision. For language models the batch is often quoted in tokens rather than examples.
- Batchingalso: Continuous batchingInference
Serving several requests in one forward pass so that each weight read from memory does work for all of them. It turns bandwidth-bound decode into higher aggregate throughput, until the KV cache runs out of memory or the batch becomes compute-bound; continuous batching adds and removes requests mid-run.
- BenchmarkEvaluation
A fixed set of tasks with a scoring rule, used to compare models: MMLU-Pro, GPQA, SWE-bench Verified, Aider polyglot, LiveCodeBench, Terminal-Bench. A published score is a claim by whoever ran it, under their settings; the course's rule is to say who reported it, and to reproduce it before relying on it.
- Benchmark contaminationEvaluation
Benchmark questions or their answers being present in a model's training data, so that a score measures recall rather than ability. Hard to rule out for public benchmarks, and the reason private test sets and fresh tasks such as LiveCodeBench's dated problems exist.
- BF16also: bfloat16Formats and quantisation
Brain floating point, sixteen bits with the exponent range of FP32 and fewer digits of precision. The working format for training and for unquantised inference on modern hardware: two bytes per parameter, and the baseline every quantised format is compared against.
- Byte-pair encodingalso: BPE, SentencePieceModels
The tokeniser training algorithm that starts from bytes or characters and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary is full. Frequent words become one token, rare words several, and any byte sequence can be represented.
- Bytes per parameterFormats and quantisation
How much memory each weight occupies in a given format: four at FP32, two at BF16 or FP16, one at FP8 or Q8, a little over half at four-bit formats once block scales are counted. Multiply by the parameter count for the weights' memory; divide bandwidth by the result for a decode-speed ceiling.
- Bytes per tokenInference
The KV-cache memory one token of context costs: two (keys and values) times layers times key-value heads times head dimension times bytes per value. Multiply by the context length and the number of concurrent requests for the cache budget; Part 4 works the examples.
C
- Chain ruleMachine learning
The calculus rule that the derivative of a composition is the product of the derivatives of its parts. Because a network is a composition of layers, the chain rule lets backpropagation reuse each layer's local derivative instead of differentiating the whole model from scratch.
- Chat templateModels
The exact text format, with special tokens, that an instruct model was trained to see conversations in: system, user and assistant turns in a fixed order. It ships in the tokenizer configuration and is applied by the engine; the wrong template is the commonest cause of a good model giving bad answers.
- CheckpointTraining
A saved copy of a model's parameters, and during training often the optimiser state and step count too. Part 1's lab saves one on every validation improvement; released models are checkpoints in Safetensors or GGUF form.
- Coder modelModels
A model post-trained on code and software tasks, often with fill-in-the-middle and tool-calling support: Qwen3-Coder, Devstral. The right choice for the coding-agent parts, and evaluated by different benchmarks than a general model.
- Compute-boundHardware
A workload whose speed is limited by arithmetic rather than by fetching bytes. Prefill and training are compute-bound because many tokens reuse each weight once it is loaded. Contrast Bandwidth-bound.
- config.jsonModels
The file in a model repository that states the architecture: hidden size, layer count, attention and key-value head counts, vocabulary size and maximum position. Everything the memory-budget arithmetic needs is in it, and Part 2's lab reads one.
- Context windowalso: Context lengthModels
The maximum number of tokens a model can attend to at once, prompt and output together. Set by training and position encoding, limited in practice by KV-cache memory, and often reduced by the engine's default; what happens when it is exceeded is the subject of a reality check in Part 7.
- Cosine similarityModels
The cosine of the angle between two embedding vectors: one for identical direction, zero for unrelated, used to rank how similar two texts are. It measures nearness in the embedding space, which correlates with meaning but is not the same thing as relevance or truth.
- Cross-entropyMachine learning
The loss used whenever a model predicts a choice among categories, including the next token of a language model: it charges more the less probability the model gave to the correct answer. A training log that prints "loss 2.31" is printing average cross-entropy over a batch.
D
- Data leakageMachine learning
Information from the evaluation data reaching the model or the decisions about it: duplicates across splits, test questions in the pretraining corpus, or hyperparameters tuned on the test set. Leakage inflates scores silently, and Benchmark contamination is its large-scale form.
- Decodealso: Generation, Token generationInference
The second phase of generation: producing one token at a time, each step reading every active weight and the whole KV cache. Bandwidth-bound at batch one, so its speed is set by memory bandwidth divided by bytes read per token.
- Decoder-onlyModels
A transformer in which each token attends only to earlier tokens, so the same model that scores a text can generate it one token at a time. All of the generative open-weight models in the course are decoder-only; embedding models are often encoders.
- Dense modelModels
A model in which every parameter is used for every token, so active and total parameters are the same. Simpler, and the best quality per byte of memory; slower to decode than a mixture-of-experts model with the same total size.
- Distillationalso: Knowledge distillationDistillation and compression
Training a small student model to reproduce a large teacher's outputs, either its token probabilities or its generated text, so that the student captures behaviour it could not have learned as well from the raw data alone. Part 15 distils a Qwen3 teacher into a Qwen3 student.
E
- Early stoppingMachine learning
Keeping the checkpoint from the epoch where validation loss was lowest, rather than the last one. The simplest and most reliable regulariser; Part 1's lab does it with a file saved on every improvement.
- Embeddingalso: Vector representationModels
A vector of numbers that represents a token, a sentence or a document, positioned so that similar meanings are near each other. Token embeddings are the model's first layer; embedding models produce a vector for a whole text and are the basis of retrieval, memory and semantic search.
- Embedding modelModels
A model trained to output one vector per text rather than to generate tokens, such as Qwen3-Embedding. Much smaller and faster than a generative model, and paired with a reranker for retrieval. A generative model's hidden states can be used the same way, less well.
- EpochMachine learning
One complete pass through the training set. Small datasets are trained for several epochs; pretraining corpora are so large that runs are measured in steps and tokens instead, since a single pass is already more than most models get.
- Evaluationalso: EvalEvaluation
Measuring what a model does on a defined task with a defined scoring rule, before and after any change: the discipline that separates "my fine-tune helped" from a feeling. Part 16 builds an evaluation harness; every hands-on page records a result in the lab notebook.
- Experiment trackingTraining
Recording, for every run, the code, data, settings, versions and results so that any number can be traced to what produced it and reproduced. The course does it with the lab notebook; Part 11 adds proper tooling for training runs.
F
- Feed-forward blockalso: MLP block, FFNModels
The second half of every transformer layer: two or three linear layers with a non-linearity between, applied to each token independently. It holds most of a dense model's parameters, and in a mixture-of-experts model it is the part that is split into experts.
- Fill-in-the-middlealso: FIMModels
A training format and prompt style in which the model is given the text before and after a gap and completes the gap, used for inline code completion in editors. Supported by coder models that were trained with the special tokens for it.
- Fine-tuningTraining
Continuing to train an existing model on new data, usually far less than it was pretrained on, to change its behaviour, style or domain. Full fine-tuning updates every weight; adapter methods such as LoRA update a small addition and fit on a home machine.
- FLOPsalso: TFLOPS, FLOPHardware
Floating-point operations, the unit of arithmetic work; a chip's peak is quoted in trillions per second (TFLOPS). Training cost is roughly six FLOPs per parameter per token. For generating text at home, Memory bandwidth usually matters more than peak FLOPs.
- Forward passMachine learning
Running inputs through the model to produce outputs, layer by layer. Inference is a forward pass alone; training keeps every layer's intermediate activations from the forward pass so the backward pass can use them, which is the main reason training needs so much more memory.
- FP16also: half precision, float16Formats and quantisation
Sixteen-bit floating point with more precision and less range than BF16. Small gradients underflow to zero in FP16 unless the loss is scaled up first, which is what loss scaling in automatic mixed precision does. Common for inference on hardware without BF16.
- FP32Formats and quantisation
Thirty-two-bit floating point, four bytes per value, about seven significant digits. The format of classical numerical computing and of optimiser state during training; rarely used to store or run a model now, because BF16 gives the range training needs at half the size.
- FP8Formats and quantisation
Eight-bit floating point, one byte per parameter, computed natively by recent NVIDIA generations. Half the size of BF16 for weights and activations; used for inference and increasingly for training on hardware that supports it.
G
- Generalisationalso: GeneralizationMachine learning
How well a model performs on examples it was not trained on. It is the only thing that matters in practice and the only thing a training loss cannot measure, which is why every evaluation in the course uses data the model never saw.
- GGUFFormats and quantisation
The single-file model format of llama.cpp: tensors, tokeniser and metadata together, with the quantisation type recorded in the file name (Q4_K_M, Q8_0). What Ollama, LM Studio and llama-server load, and what Part 6's quantisation lab produces.
- GradientMachine learning
For each parameter, how much the loss changes when that parameter is nudged: the slope of the loss landscape under the current parameters. Backpropagation computes it for every parameter at once, and gradient descent steps against it.
- Gradient descentMachine learning
The procedure that trains every model in this course: compute the gradient of the loss, move every parameter a small step in the opposite direction, repeat. Adam, AdamW and momentum are refinements that change the size and steadiness of the steps, not the direction they follow.
- Grouped-query attentionalso: GQA, Multi-query attentionModels
Sharing each key-value head among a group of query heads, so the KV cache holds far fewer heads than the model computes queries for. Most current open models use it; it is why a model's
num_key_value_heads, not itsnum_attention_heads, sets the cache size per token.- GRPOAlignment and reinforcement learning
Group relative policy optimisation: sample several responses to one prompt, score each, and push the model towards the ones that scored above the group's average. It needs no value model, which is why reinforcement learning on a single home machine is practical.
H
- HallucinationModels
Fluent output that is not true. It follows from the objective: the model is trained to produce likely continuations, and a plausible fabrication is often a likely continuation. Grounding, retrieval and verification reduce it; no setting removes it.
- HeadroomHardware
Memory deliberately left unused when planning a model deployment: the operating system, the engine's buffers, the display, other applications, and the KV-cache growth of a long conversation. Filling memory to the last gigabyte is the commonest reason a model that "fits" crashes or slows down.
- Hugging Face Hubalso: Hugging FaceModels
The repository where open-weight models, datasets and tokenisers are published, downloaded with the
hfcommand-line tool or thehuggingface_hublibrary. A model repository holds its weights, config, tokeniser files and model card.- Hybrid architectureModels
A model that mixes attention layers with layers of another kind, state-space or linear-attention blocks such as Mamba-2 or gated DeltaNet, to cut the KV cache and the quadratic cost of long contexts. Qwen3-Next and Nemotron 3 Nano are the course's examples.
- HyperparameterMachine learning
A setting chosen by the person rather than learned from data: the learning rate, batch size, number of epochs, model width and depth, LoRA rank. Hyperparameters control the search for parameters and are the first suspects when a training run misbehaves.
I
- InferenceInference
Running a trained model to get outputs: a forward pass with no gradients and no optimiser. For a language model it is prefill of the prompt followed by decode of the output one token at a time, and everything in Parts 5 to 10 is about doing it well on local hardware.
- Instruct modelalso: Chat modelModels
A model post-trained to follow instructions and hold a conversation in a chat template; the variant most readers want, marked "Instruct", "it" or "Chat" in the name. It answers rather than continues, and expects its template.
K
- KV cacheInference
The keys and values of every token processed so far, kept in memory so that generating the next token does not recompute them. It grows linearly with context and is the reason long contexts cost memory: bytes per token times tokens, per concurrent request.
L
- Lab notebookalso: labbookTraining
The single file,
labbook.mdin the course directory, that every lab appends a result to: machine, versions, settings, number, date. The course's discipline of measurement lives in it, and the capstone is written from it.- Layer normalisationalso: RMSNorm, LayerNormModels
Rescaling a token's vector to a standard size before attention and before the feed-forward block, so that activations stay in a range the next layer expects. RMSNorm is the cheaper variant most open models use.
- Learning rateMachine learning
How far each gradient-descent step moves the parameters. Too large and the loss bounces or explodes; too small and training crawls. Schedules start it high, after a warmup, and lower it over the run, usually with cosine decay.
- Licencealso: License, Apache-2.0, MIT licenceSecurity and licensing
The terms under which a model's weights may be used, modified and redistributed. Apache-2.0 and MIT are permissive; the Llama community licence and the Gemma terms add conditions and use restrictions; some models carry a publisher's own text. The course's rule is that every model named carries its licence.
- llama.cppInference
The C++ inference engine, with its
llama-server,llama-cliandllama-benchtools, that runs GGUF models on CPUs and on every GPU in the course through CUDA, Vulkan, Metal and ROCm backends. Ollama and LM Studio are built on it. Part 6 is about it.- LoRAalso: QLoRA, AdapterTraining
Low-rank adaptation: freezing the model's weights and training small low-rank matrices added to some of them, cutting the trainable parameters and optimiser memory by orders of magnitude. The method that makes fine-tuning an 8B model possible on a 16 GB machine; QLoRA adds a quantised base.
- Loss functionalso: LossMachine learning
A function that turns a model's prediction and the correct answer into one number, zero for a perfect prediction and larger the worse it is. Training minimises the average loss over the training examples. Language models use Cross-entropy.
M
- Matrix multiplicationalso: Matmul, GEMMHardware
The operation that is almost all of a model's work: multiplying an activation matrix by a weight matrix. Every one of its multiply-adds is independent, which is what a GPU parallelises and what tensor cores do in blocks. Its cost is what FLOPs count.
- Memory bandwidthHardware
How many bytes per second a processor can read from its memory. Decoding a token reads every active weight once, so the decode speed of a model is close to bandwidth divided by the bytes of weights read per token. The single most useful number about a machine for local inference.
- Memory budgetalso: What fits whereHardware
The arithmetic that says whether a model fits a machine: weights at their bytes per parameter, plus the KV cache for the intended context and concurrency, plus the engine's working memory, plus headroom for the operating system and other applications. Part 4 works it and Part 5 measures it.
- Memory hierarchyHardware
Registers and cache on the chip, then accelerator memory, then system memory, then storage, each larger and slower than the last. A model must fit in the level the accelerator reads at full speed; anything below that level is the slow path.
- Mixed precisionalso: AMP, Automatic mixed precisionTraining
Training with most arithmetic in BF16 or FP16 while keeping a master copy of the parameters and the optimiser state in FP32. It roughly halves activation memory and speeds up tensor-core arithmetic; PyTorch's
torch.ampimplements it, with loss scaling for the FP16 case.- Mixture-of-expertsalso: MoE, Sparse modelModels
An architecture whose feed-forward blocks are split into many experts, with a router choosing a few per token. The whole model must sit in memory but only the active experts are read per token, so it decodes like a small model and knows like a large one: the design that suits unified-memory machines.
- MNISTMachine learning
The classic dataset of seventy thousand handwritten digits, twenty-eight by twenty-eight pixels each, used in Part 1's lab because it downloads in seconds and a two-layer network learns it in a minute. Its only job in the course is to make the training loop visible.
- Model cardSecurity and licensing
The document published with a model: what it is, how it was trained, what it was evaluated on, its licence and its intended and prohibited uses. The first thing to read about any model, and the source of every "reported by" number the course quotes.
- Model familyModels
A publisher's line of models released together or in sequence under one name, at several sizes and variants: Llama, Qwen, Gemma, gpt-oss, DeepSeek, Mistral and the rest. Families share a tokeniser and architecture across sizes, which is what makes same-family distillation easy.
- Model libraryTraining
One directory on disk, organised by publisher and model, that every engine and tool on the machine reads from, so that a model is downloaded once and shared. Part 4's lab creates it and the later parts point llama.cpp, vLLM and the agent tools at it.
- Multi-head attentionalso: Attention headModels
Running several attention operations in parallel, each with its own smaller query, key and value projections, so different heads can attend to different kinds of relationship. Grouped-query attention lets several query heads share one key-value head to shrink the cache.
- MXFP4Formats and quantisation
A four-bit floating-point block format from the Open Compute Project's microscaling family: groups of values share one scale factor. The native weight format of the gpt-oss models, so those checkpoints are already about half a byte per parameter without a separate quantisation step.
N
- Neural networkMachine learning
A function built from layers of weighted sums followed by non-linearities, with parameters found by gradient descent. Width is the number of units in a layer; depth is the number of layers. A transformer is a neural network with a particular arrangement of layers.
- Next-token predictionalso: Language-modelling objective, AutoregressiveModels
The one objective every language model is trained on: given the tokens so far, output a probability for every token in the vocabulary being next. Chat, code and reasoning are all this objective applied repeatedly; hallucination is a consequence of it, not a bug in it.
- NVFP4Formats and quantisation
NVIDIA's four-bit floating-point format with a shared scale per small block of values, computed natively by the Blackwell generation including the DGX Spark. Half a byte per parameter. MXFP4 is the related open block format that gpt-oss ships its weights in.
O
- Offloadingalso: CPU offloadInference
Keeping part of a model or its KV cache in slower memory, system RAM or disk, and moving pieces to the accelerator as needed. It lets a model that does not fit run at all, at a decode speed set by the slow link. On unified-memory machines there is nothing to offload to.
- Open sourceSecurity and licensing
Software or, under the Open Source Initiative's AI definition, a model whose weights, code and enough data information are available under a licence that permits use, study, modification and sharing for any purpose. Apache-2.0 and MIT qualify; the Llama and Gemma terms do not.
- Open weightsalso: Open-weight modelSecurity and licensing
A model whose trained parameters are published for download under some licence, whatever its terms. It says nothing about the training data or code and is not the same as open source: the licence decides what you may do, and this course names it beside every model.
- Optimiseralso: Optimizer, Adam, AdamWMachine learning
The algorithm that applies gradients to parameters: plain stochastic gradient descent, or Adam and AdamW, which scale each parameter's step by the history of its gradients and are the default for language models. The optimiser keeps its own state per parameter, which is why training needs several times the memory of inference.
- OverfittingMachine learning
When a model fits the training examples better than it fits new examples: it has memorised accidents of the training set instead of the pattern behind it. Visible as training loss that keeps falling while validation loss stops falling and rises.
P
- ParameterMachine learning
One of the adjustable numbers inside a model: a weight or a bias. Training searches for parameter values that make the loss small; a model's size is usually quoted as its parameter count, so an 8B model has about eight billion of them. Compare Hyperparameter, which is set by the person, not learned.
- Parameter countalso: Model sizeModels
The number of learnable values in a model, the "8B" in a name: embedding and output matrices, every attention and feed-forward weight, and for mixture-of-experts models every expert, whether or not it is active for a given token.
- PerplexityEvaluation
The exponential of the average cross-entropy per token: how surprised the model is by a text, with lower being better. A standard way to compare a quantised model with its original on the same text, and the metric Part 16 uses first.
- Post-trainingTraining
Everything done to a base model after pretraining to make it useful: supervised fine-tuning on demonstrations, preference tuning, reinforcement learning. It changes behaviour and format far more than knowledge, which is set by pretraining.
- Post-training quantisationalso: PTQ, GPTQ, AWQFormats and quantisation
Quantising a finished model without further training: rounding its weights to a lower-bit grid with per-block scales, optionally calibrated on sample text so that the most important weights are rounded least. What GGUF conversion, GPTQ and AWQ do; Part 16 runs and measures it.
- Preference leaderboardalso: LMArena, Elo ratingEvaluation
A ranking built from people choosing between two anonymous models' answers to their own prompts, such as LMArena. It measures what users prefer, which includes style and confidence, and is not the same as correctness; useful as one signal among several.
- Preference optimisationalso: DPO, Direct preference optimisationAlignment and reinforcement learning
Training a model directly on pairs of a preferred and a rejected response, so it raises the probability of the former relative to the latter, without a separate reward model or a reinforcement-learning loop. DPO is the original; Part 14 runs it on a home machine.
- Prefillalso: Prompt processingInference
The first phase of generating a response: processing every prompt token at once, filling the KV cache. It is a large matrix multiplication and therefore compute-bound; its duration is the time to first token.
- PretrainingTraining
Training a model from random initialisation on a web-scale corpus with the next-token objective, for weeks on thousands of accelerators. It produces a base model that knows language and much of what was written down, and nothing about following instructions.
- PruningDistillation and compression
Removing weights, heads, or whole layers from a trained model and then retraining briefly to recover, producing a smaller model from a larger one. Used by publishers to make small variants of big models; combined with distillation to recover the lost quality.
Q
- Quantisationalso: QuantizationFormats and quantisation
Storing weights, and sometimes activations, in fewer bits than they were trained in, to cut memory and speed up bandwidth-bound decoding. Post-training quantisation rounds an existing model; quantisation-aware training trains with the rounding in the loop. The quality cost is small, real and measurable.
- Quantisation-aware trainingalso: QATFormats and quantisation
Training or fine-tuning with the quantisation rounding simulated in the forward pass, so the model learns weights that survive rounding. Costs a training run; produces better low-bit models than post-training quantisation, which is why some publishers release quantisation-aware variants.
- Queries, keys and valuesalso: QKVModels
The three vectors attention computes for every token from its residual-stream state. A token's query is compared with every earlier token's key to produce attention weights, which then mix those tokens' values. Keys and values are what the KV cache stores.
R
- Reasoning modelalso: Thinking modelModels
A model post-trained, usually with verifiable rewards, to write out a chain of intermediate steps before its answer. Better at maths, code and multi-step tasks, at the cost of many more output tokens; "thinking" modes and budgets control how much it writes.
- Reinforcement learning with verifiable rewardsalso: RLVRAlignment and reinforcement learning
Reinforcement learning where the reward is computed by a checker rather than a learned model: a unit test passes, a maths answer matches, a format is valid. The training method behind reasoning models, and the one Part 14 applies with GRPO to a small model.
- RerankerModels
A model that scores how well a candidate document answers a query by reading both together, used after an embedding search to reorder its top results. Slower per pair than an embedding comparison, more accurate, and applied only to a short list.
- Residual connectionalso: Skip connectionMachine learning
A shortcut that adds a layer's input to its output, so each layer learns a correction rather than a whole new representation. Residual connections are why networks can be dozens of layers deep and still train; in a transformer the running sum is called the Residual stream.
- Residual streamModels
The running vector, one per token, that every transformer layer reads from and adds its result back into. Because layers add to it rather than replace it, the stream carries information from the embedding to the output and makes deep stacks trainable.
- Retrieval-augmented generationalso: RAGInference
Fetching relevant documents with an embedding search and placing them in the prompt so that the model answers from them rather than from memory alone. It reduces hallucination on facts the documents cover and is taught in Part 10 and used by the agent systems of Part 26.
- Reward modelAlignment and reinforcement learning
A model trained on human preference comparisons to score how good a response is, used as the training signal in reinforcement learning from human feedback. It is a learned proxy for preference and can be gamed, which is why verifiable rewards replaced it where they can.
- RLHFAlignment and reinforcement learning
Reinforcement learning from human feedback: training a policy to maximise a reward model's score of its responses, with a penalty for drifting from the fine-tuned starting point. The method behind the first assistant models; direct preference optimisation is its cheaper successor.
- RoPEalso: Rotary position embedding, Position encodingModels
Rotary position embedding: encoding a token's position by rotating its query and key vectors by an angle that depends on the position, so attention scores depend on relative distance. Scaling tricks on RoPE are how models are extended to longer contexts after training.
S
- SafetensorsFormats and quantisation
Hugging Face's checkpoint format: a header describing every tensor's name, dtype and shape, followed by the raw bytes, loadable without executing code. The release format of almost every open-weight model, usually in BF16, and what transformers, vLLM and SGLang load.
- Samplingalso: Top-p, Top-k, Nucleus samplingInference
Turning the model's probability distribution over the next token into one chosen token: greedy takes the most likely, temperature reshapes the distribution, top-p and top-k cut off its tail. The same model with different sampling settings writes differently.
- Scaling lawsalso: ChinchillaTraining
Empirical relationships between loss and the amount of compute, data and parameters: loss falls smoothly and predictably as all three grow. The Chinchilla result that data and parameters should grow together is why modern models are trained on far more tokens per parameter than older ones.
- Special tokenModels
A vocabulary entry with a role rather than text: beginning and end of sequence, the boundaries of a chat turn, padding, tool-call markers. The chat template inserts them, and a prompt sent without them makes an instruct model behave like a base model.
- State-space modelalso: Mamba, SSM, Linear attentionModels
A sequence layer that carries a fixed-size state from token to token instead of attending over all previous tokens, so its cost per token and memory do not grow with context. Mamba and Mamba-2 are the common designs; in practice they appear in hybrids alongside some attention layers.
- Supervised fine-tuningalso: SFTTraining
Training a model on example conversations or completions with the ordinary next-token loss, so it imitates the demonstrations. The first post-training stage and the technique Part 13 teaches with LoRA on a reader's own data.
- Synthetic dataDistillation and compression
Training examples generated by a model rather than collected from people: a teacher writing questions and answers, solving problems, or rewriting text. Most of what distillation and much of what post-training use; its quality decides the student's.
T
- TemperatureInference
A sampling setting that flattens or sharpens the next-token distribution before a token is drawn: near zero is close to always taking the most likely token, higher values make rarer tokens more likely and the output more varied and less reliable.
- TensorHardware
An array of numbers with a shape: a vector, a matrix, or a higher-dimensional block. Every quantity in a model is a tensor, operations are defined on shapes, and "shapes do not match" is the first error message every practitioner learns to read.
- Tensor coreHardware
A unit in NVIDIA GPUs that performs a small block of matrix multiplication as a single operation, at lower precision than the general cores. Mixed-precision training and FP8 and FP4 inference exist largely to feed tensor cores the formats they are fastest at.
- Test setMachine learning
Examples looked at once, at the end, with the model chosen on the validation set. It is the honest measure of generalisation; reuse it to make decisions and it silently becomes a second validation set.
- Thinking modealso: Reasoning budgetModels
A switch, in models such as Qwen3, between producing reasoning tokens before the answer and answering directly, sometimes with a budget on how many reasoning tokens to spend. It trades latency and output tokens for accuracy on hard tasks.
- Time to first tokenalso: TTFT, Time per output token, TPOTInference
The delay between sending a prompt and receiving the first output token: prefill time plus queueing. It grows with prompt length and is what makes a long-context request feel slow even when decode is fast. Time per output token is the decode-side counterpart.
- TokenModels
The unit a language model reads and writes: a piece of text, often a word fragment, chosen by the tokeniser from a fixed vocabulary. Context windows, prices, speeds and memory are all counted in tokens, and the same text is a different number of tokens in different models and languages.
- Tokeniseralso: TokenizerModels
The component that splits text into tokens and maps each to an integer id, and back again. Byte-pair encoding and SentencePiece are the common algorithms; the tokeniser is part of the model, and two models with different tokenisers cannot share a checkpoint or a KV cache.
- Tokens per secondalso: tok/s, ThroughputInference
The rate of decode, quoted per request or aggregated over a batch. Meaningless without the machine, engine, model, quantisation, context length and batch size that produced it, which is why this course puts every such figure in a table with that context and never in prose.
- Tool callingalso: Function callingInference
A model producing a structured request to run a function, receiving the result, and continuing; the mechanism behind agents. Models are post-trained for it and engines expose it through the chat API; Part 24 teaches it in depth.
- Training loopMachine learning
Batch, forward pass, loss, backward pass, optimiser step, repeated. Every training run in the course, from the MNIST classifier to a fine-tune of an eight-billion-parameter model, is this loop; the frameworks hide it behind a call but it is always there.
- Training setMachine learning
The examples the model's parameters are fitted to. Loss on the training set says how well the model reproduces what it was shown, nothing more; it must be kept separate from the Validation set and the Test set.
- TransformerModels
The architecture of every model in the course: an embedding, a stack of identical layers each made of attention and a feed-forward block around a residual stream, and an output projection to vocabulary probabilities. Decoder-only transformers generate text one token at a time.
U
- Unified memoryHardware
One pool of memory shared by the CPU and the GPU with no copying between them, as on the DGX Spark, the Ryzen AI Max+ machines and Apple silicon. It lets very large models fit on a desk-sized machine; its cost is system-memory bandwidth, lower than a discrete card's VRAM.
V
- Validation setalso: Dev set, Held-out setMachine learning
Examples held out from training and used during a run to choose hyperparameters and to decide when to stop. Because decisions are made from it, its score drifts optimistic, which is why a separate Test set is kept for the final number.
- Vision-language modelalso: VLM, Multimodal modelModels
A model that accepts images as well as text, by encoding image patches into tokens the language model attends to. Qwen3-VL is the course's example, used for screenshots and documents in the agent parts; it needs an engine that supports its image encoder.
- VocabularyModels
The fixed set of tokens a model can read or write, typically 32,000 to 260,000 entries. Its size sets the width of the embedding and output matrices, which is why vocabulary is a noticeable share of a small model's parameters.
- VRAMalso: GPU memory, Video memoryHardware
The memory on a discrete graphics card, fast and limited. A model must fit in VRAM to run at full speed; weights kept in system memory are reached over PCIe and the decode speed collapses. Contrast Unified memory.
W
- WeightMachine learning
A parameter that multiplies an input. In a neural network almost every parameter is a weight in a matrix, which is why matrix multiplication dominates the work and why a model's memory footprint is its weights times the bytes per weight.