Skip to content
Local LLM Course

What a Language Model Is

A language model is a file full of numbers and a small amount of code that multiplies them together. Everything that surprises people about these systems, that they invent citations, that long conversations get slower and hungrier for memory, that a model with thirty billion parameters can generate as fast as one with three, follows from what those numbers are and how they are used.

This part opens the box. It teaches the single objective every model in this course was trained on, the pieces of text it actually sees, the geometry it turns them into, the mechanism that lets each piece look at the others, and the arithmetic of counting a model. Then you download a small model and check every claim on your own machine.

The reference models are the ones the rest of the course uses, so the numbers here are the numbers you will meet again: Qwen3-8B and Qwen3-1.7B for the dense case, gpt-oss-20b and Qwen3-30B-A3B for mixture-of-experts, and Llama 3.1 8B as the cross-engine comparison. Their sizes and licences are on the model reference.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • state the training objective in one sentence and explain why hallucination is a consequence of it rather than a defect bolted on afterwards;
  • explain why a model sees tokens rather than words, predict roughly how a given piece of text will be split, and say why the same sentence costs more tokens in Portuguese than in English;
  • describe what an embedding is, what cosine similarity measures, and what it does not imply;
  • draw a transformer block from memory, name what attention computes, and say why its cost grows faster than the length of the context;
  • read a config.json and work out the layer count, the head counts, the vocabulary size and the size of the embedding matrix, and explain the difference between total and active parameters;
  • open a checkpoint, count it, tokenise with it, run one forward pass and read the probabilities the model assigns to the next token.

What you need first

Section titled “What you need first”

Part 1. In particular the loss-and-gradients lesson, because the training objective here is a loss function, and the tensors-and-precision lesson, because everything in this part is a tensor with a shape and a bytes-per-parameter cost. The lab assumes the Python environment and the lab notebook that Part 1’s lab created.

The lab downloads Qwen3-1.7B, which is under 4 GB on disk in its native precision and runs on the 8 GB tier on all four platform tracks, on the CPU if there is no accelerator. Nothing here needs a large machine.

How to work through it

Section titled “How to work through it”

Read the five lessons in order. The vocabulary compounds: tokens are the input to embeddings, embeddings are the input to attention, attention and the feed-forward block are what the layer count counts, and the last lesson adds them up into a file size and a memory budget.

Then do the lab with a terminal open. It is the first time in the course that you handle a real checkpoint, and it is deliberately unglamorous: four short scripts that print tensor shapes, token ids, probabilities and attention weights. Every later part refers back to what those scripts show. Part 3 then explains how models are made and improved, and Part 4 turns the arithmetic in this part into a way of choosing which model fits your machine.

Part2 of 27Level1 — AI LiteratePages6Estimated time3.4 hours

0 / 6 lessons in this part completed