Skip to content
Local LLM Course

Putting Models to Work

You now have an engine, a model library and a front-end. What you do not yet have is a reason to trust anything you build on top of them. This part is about the difference between running a model and getting work out of one, and about the measurement that tells you which is which.

Six lessons, a project and a lab. The lessons cover the things that decide whether a local model is useful: the formatting it expects, how to make it return a schema instead of a paragraph, how to put it in your editor, how to give it your documents, what it costs to send it an image or an hour of audio, and what to do before its endpoint is reachable from anything other than localhost. The project builds a question-answering service over your own files, with citations and a refusal path. The lab builds the evaluation set that Level 3 uses every time it claims a model got better.

What you will be able to do

Section titled “What you will be able to do”

By the end of this part you should be able to:

  • explain what a chat template is, name what breaks when the wrong one is applied, drive a model’s thinking mode deliberately, and set the sampling parameters its own card recommends rather than the ones your engine defaults to;
  • get a validated JSON object out of a local model on every request, using a schema in the request, a grammar in the sampler, and a validation-and-retry loop that fails loudly instead of quietly;
  • run completion and chat from a local model inside your editor, work out from your own hardware whether the completion will arrive fast enough to be worth having, and prove from the server’s log that nothing left the machine;
  • build a retrieval pipeline over your own documents: chunk, embed, index, retrieve, rerank and answer with citations, and name the four ways it produces a confident wrong answer;
  • send an image, a scanned page or a recording to a local model on your track, price the memory that costs, and say which engines support which modality;
  • state what a served endpoint exposes, bind and authenticate it, decide what to log and for how long, and check a downloaded model for the two supply-chain problems that matter;
  • keep a personal task set with reference answers, score a model against it with a judge model, and describe the three biases that make a judge score disagree with your own.

What you need first

Section titled “What you need first”

Everything in Level 2. Part 6 supplies llama-server, which is the endpoint every script here talks to. Part 7 supplies Ollama or LM Studio and the Open WebUI front-end that the project can optionally reuse. Part 4 supplies the ~/models library and the memory arithmetic. Part 1 supplies the Python environment and labbook.md, which by the end of this part will hold your first evaluation results.

Part 9 builds a gateway: LiteLLM in front of llama-swap, one endpoint with one key that loads models on demand. Every script in this part takes the server address on the command line, so it runs against that gateway, against a plain llama-server, or against anything else that speaks the OpenAI-compatible API. If you have not done Part 9 yet, use http://127.0.0.1:8080/v1 and carry on.

The project and the lab both run on the 8 GB tier. The retrieval work uses two 0.6B-class models alongside a 4B or 8B chat model, which is a comfortable fit everywhere. Nothing here needs a second machine.

How to work through it

Section titled “How to work through it”

Read the prompting lesson first and do not skip it, even if you have been prompting hosted models for a year. Almost every “this local model is bad” report has a formatting problem underneath it, and the lesson is mostly about formatting.

Structured output and coding assistants are independent of each other; read whichever you will use sooner. Retrieval is the longest lesson and the one the project is built from, so read it immediately before the project. The multimodal and security lessons stand alone; the security lesson is the one to read before you show anybody else what you have built.

Do the project before the lab if you want something to show, and the lab before the project if you want to know whether your model is good enough to build on. The lab is the more important of the two: Level 3 is a sequence of changes to a model, and without a task set of your own, every “it improved” in it is a feeling rather than a measurement.

Part10 of 27Level2 — Local OperatorPages8Estimated time5.2 hours

0 / 8 lessons in this part completed