Lab Execution and Evidence Guide

Prepare once, then follow the selected lab

Labs are procedures for you to execute and validate on your own hardware. Their “Validated on” line distinguishes documentation-derived instructions from a recorded hardware run. A website build, a syntax check or a successful package installation is not hardware validation. Treat timings in requirements as planning estimates; record your own attended, download and compute time separately.

Use Bash for shell examples, including on macOS. On Windows, use the WSL environment named in your track. A command shown as output is something to inspect, not something to paste. A fragment requires substitution or surrounding configuration; read the paragraph before it.

Get a complete, consistent copy of the files

The simplest path is a repository checkout. This includes helper modules and sample documents that individual download buttons might not make obvious. The following creates a checkout in a dedicated directory; if you already have one, set COURSE_REPO to its absolute path and skip the clone.

RunnableAll tracks

mkdir -p "$HOME/llm-course"
git clone https://github.com/ebrandi/llm-course.git "$HOME/llm-course/course"
export COURSE_REPO="$HOME/llm-course/course"
git -C "$COURSE_REPO" rev-parse HEAD

Expected result: Git prints a commit identifier. Save it in your notebook. If the destination already exists, do not remove it to make the command succeed: use that checkout or choose a fresh directory. For a downloaded source archive, extract it and set COURSE_REPO to the directory containing package.json, scripts/ and src/.

Copy the lab files into a separate execution area. This helper uses Python’s standard library, preserves the part directories, includes sample documents and places shared imported helpers beside the scripts that require them. It downloads nothing and starts no services.

RunnableAll tracks

export LABS_ROOT="$HOME/llm-course/labs"
python3 "$COURSE_REPO/scripts/prepare-lab-workspace.py" --destination "$LABS_ROOT"

Expected result: Prepared ... files followed by the destination. course-workspace.json records the source commit and file hashes. Running the helper again preserves identical files. If files have changed, it stops before copying; choose another destination for a new course revision and carry your datasets and results forward deliberately. It does not overwrite your edited lab scripts.

Each hands-on lesson names its part directory. Set LAB_DIR and change into it before running bare filenames such as python3 train-sft.py. Sibling paths such as ../part-24-tools-mcp-and-the-agent-loop then resolve correctly. Do not flatten all parts into one directory: several parts intentionally contain different files called compose.yaml, env-example.txt or report-template.md.

Individual website downloads are also usable: download all files listed by the lab, including supporting modules and data, preserving directories. Use the download index for files inherited from prerequisite lessons. The repository workspace is the recommended route when following the entire course.

Establish the terminal and environment contract

Use one terminal for each foreground server and another for the client or measurement script. Label them by role and port in your notebook. A server command that keeps running has not hung merely because it does not return to the prompt. Wait for its readiness log, then use the client terminal. Stop it with Ctrl+C in the server terminal when the procedure says to change configurations.

Environment variables belong to a shell session. A newly opened terminal needs COURSE_REPO, LABS_ROOT, LAB_DIR, the selected virtual environment and any lesson-specific variables set again. Do not assume that exporting a URL or model path in one terminal exports it in another.

Before an expensive step, inspect the basics:

RunnableAll tracks

pwd
command -v python3
python3 --version
python3 -m pip --version
df -h .

The directory should be the lesson’s working directory, and Python and pip should belong to the environment selected in its platform instructions. If the track uses a container, run the Python checks inside that container. A successful package import on the host does not check the container.

Do not install a generic accelerator wheel over a vendor-provided stack. Follow the platform’s environment lesson, then return here. Serving and training may use different virtual environments; record which terminal has which environment active.

Substitute configuration without guessing

An endpoint root and an API base URL are different strings. For example, a server can listen at http://127.0.0.1:8080, while its client library expects http://127.0.0.1:8080/v1. Read the script’s help and use the form it asks for. Duplicating /v1 is a common cause of a route-not-found response. Inside a container, 127.0.0.1 refers to that container, not your host or another container.

Choose the model identifier from the actual endpoint’s model listing. It may be an alias rather than a repository name or a GGUF filename. Keep filesystem paths and served names in separate variables. Verify that a path exists before launching a loader. Never paste text in angle brackets literally: those marks in a fragment describe a value you must supply.

Configuration examples containing credentials need locally generated values as directed in their lesson. Keep secret values out of screenshots, command transcripts and committed evidence. A secret’s variable name can be recorded without recording the secret itself.

Execute through checkpoints

Read a task’s command, expected result and stop condition before running it. Start with the smallest useful request or training run, then increase size. After a failed command, stop at that step; downstream errors from missing files obscure the original failure.

Distinguish these failure classes:

Observation	First evidence to collect	Next action
File or module missing	Working directory and exact missing path	Check copied files and the active environment
Connection refused	Server terminal and configured host/port	Start or repair the intended server before retrying
HTTP error response	Status and response body, with secrets removed	Check route, authentication, model name and request shape
Out of memory	Phase of failure, context, batch and device placement	Use the lesson’s reduced path; change one size setting
Non-finite loss	First failing step, dtype, input batch and gradient statistics	Inspect data and numerical stability before restarting
Valid but incorrect output	Exact prompt, raw output and independent check	Investigate task behaviour; installation may already be correct

When capturing a command with a pipe to a log, use Bash’s pipefail so a successful tee does not hide a failing training command. Record the exit status immediately. Avoid a broad command trace such as set -x when arguments or environment expansions contain credentials.

Keep an evidence record that another person can replay

For each experiment, save the objective, machine and OS, package or image versions, model revision and representation, input dataset hash, command/configuration, random seed, timestamps, raw results and your interpretation. Keep estimates separate from measured values. A useful outcome can be “the change did not improve this task”; it still needs the measurement and its controls.

Use three completion states: pass, fail or not run, with a reason. A hardware-specific alternative is a separate row. Running a single-machine fallback does not validate a two-machine transport. A model answering once does not validate the lab’s load, quality or security criteria.

Retain complete task-level results when reporting an average. Include timeouts, parser failures and rejected requests in the denominator defined by the experiment. Do not keep rerunning until the desired conclusion appears. If you change settings after observing failures, give the new run a new identity and preserve the first result.

Restart and clean up deliberately

Use a fresh output directory for a new training experiment. Loading weights is not necessarily resuming an optimiser: an exact resume may also require scheduler, optimiser, random state and data position. Follow the checkpoint semantics of the script in that lesson.

Before cleanup, identify what later parts require. Raw agent trajectories feed Part 27; evaluation tasks feed model comparisons; adapters require the exact base checkpoint; the capstone uses your labbook and raw measurements. Archive these before removing temporary caches or intermediate conversions. Stop only the services you started for the lab, then repeat the relevant health probe to confirm shutdown. A cleanup command is optional when you intend to keep using the service.

For a hardware validation report, include the last completed checkpoint, the first failure if any, and the exact adjustment that allowed progress. That makes your validation useful to another learner without turning a local workaround into an unsupported general claim.