Vision, Speech and Documents: Multimodal Locally

Most of the material you would actually like a model to work with is not plain text. It is a photograph of a whiteboard, an hour of a meeting, a scanned invoice, a screenshot of an error. By the end of this lesson you will be able to send each of those to a model on your own machine, say what each one costs in context and memory before you send it, and state which engine supports which modality on your track rather than finding out by failing.

An image becomes tokens

There is one mechanism under all of this, and knowing it removes most of the surprises.

A multimodal model is a language model with an extra encoder bolted on the front. The encoder turns an image, or a slice of audio, into a sequence of vectors, a projector maps those vectors into the same space the language model’s token embeddings live in, and from that point on they are just tokens in the context. The model does not have a special image input; it has a longer prompt.

What happens to a picture on the way into the model

pixels → context

1. Your image or audioA file, a base64 data URL, or a path on the machine running the server.you supply this
2. EncoderA vision or audio tower. Separate weights from the language model, often a separate file.
3. ProjectorMaps the encoder output into the language model's embedding space. In llama.cpp this is the mmproj file.
4. ContextThe result occupies context positions exactly like text tokens, and costs key-value cache exactly like text tokens.this is the bill
5. Language modelPrefill and decode as usual. Nothing downstream knows the tokens came from a picture.

The single most useful consequence: everything you learned about context length and key-value cache in Parts 4 and 6 applies unchanged to images and audio.

Three practical facts fall out of that diagram.

An image has a token cost, and it is not small. A page-sized image at a resolution high enough to read the small print is worth a substantial fraction of a short document. Several images in one conversation add up quickly, and the failure looks like a context overflow, not like an image problem.

The encoder is extra weights. In llama.cpp it is a separate GGUF, the multimodal projector, that you load alongside the model. It has to fit in memory too.

Resolution is a dial you control. Sending a smaller image costs fewer tokens and reads less small print. For “what is on this whiteboard” a small image is fine; for “read this invoice number” it is not, and choosing per task rather than globally is the difference between a usable pipeline and an expensive one.

Qwen3-VL-8B at Q4_K_M with an 8k context, on a machine with 8 GB available to the model

Weights (Q4_K_M)

5.4 GB

KV cache, 8k context

1.2 GB

Free

1.4 GB

Total

8 GB

Weights from the course model table; key-value cache from the per-token figure in the same table multiplied by the context length. The multimodal projector and runtime overhead are not shown because their sizes vary by build, which is exactly why the 8 GB tier is tight here rather than comfortable.

Vision, by engine

llama.cpp documents multimodal input through a library it calls libmtmd, stating that “llama.cpp supports multimodal input via libmtmd” and naming the tools that use it: llama-cli, llama-server through the OpenAI-compatible chat completions API, and llama-mtmd-cli. The model and its projector can be loaded together: with -hf the projector is downloaded for you, and --mmproj points at a local projector file when you have your own. --no-mmproj-offload keeps the projector on the processor when video memory is scarce, which is a useful lever on an 8 GB card.

RunnableAll tracks

a vision model and its projector, from the hub

llama-server \
  --hf-repo ggml-org/Qwen2.5-VL-7B-Instruct-GGUF:Q4_K_M \
  --alias vision \
  --ctx-size 8192 \
  --host 127.0.0.1 \
  --port 8082

RunnableAll tracks

one image, one question, from the command line

llama-mtmd-cli \
  --model ~/models/vision/model-Q4_K_M.gguf \
  --mmproj ~/models/vision/mmproj-f16.gguf \
  --image ~/pictures/whiteboard.jpg \
  --prompt "Transcribe the text on this whiteboard. If a word is unreadable, write [unclear]."

⌚Version note

The llama.cpp multimodal page’s list of supported vision families, checked on 2026-09-08, names Gemma 3, SmolVLM, Pixtral 12B, Qwen2-VL and Qwen2.5-VL, along with omni-modal families such as Qwen2.5-Omni and Qwen3-Omni that also take audio. The Qwen3-VL card states that quantised builds are available for llama.cpp, Ollama, LM Studio and Jan. Support for a new vision family lands in llama.cpp weeks or months after the model, and the projector format sometimes changes with it, so check that page against your build before assuming a family works, and prefer a projector published beside the quantised weights you are using.

Qwen3-VL-8B-Instruct is this course’s reference vision model: Apache-2.0, a native context of 256K tokens which its card says is expandable to 1M, optical character recognition that the card describes as covering 32 languages, document structure parsing, video understanding, and operating graphical interfaces well enough to be used as a screen agent, which is where Part 26 picks it up.

vLLM takes multimodal data in two shapes. Offline, a multi_modal_data dictionary alongside the prompt, keyed "image", "audio" or "video". Online, through the OpenAI-compatible chat API, where an image is a content item of type image_url whose URL may be a base64 data URL, audio arrives as input_audio or audio_url, and video as video_url. --limit-mm-per-prompt caps how many items of each modality a request may carry, which is a control worth setting before anybody else can reach the endpoint.

MLX has mlx-vlm, described as “a package for inference and fine-tuning of Vision Language Models (VLMs) and Omni Models (VLMs with audio and video support) on your Mac using MLX”, MIT-licensed, installed with pip install -U mlx-vlm. It provides a generation command, a server and a chat interface, and lists Qwen among more than thirty supported architectures.

Speech to text

Two families matter for local work, and the course pins one of them.

Parakeet is the course’s reference. parakeet-tdt-0.6b-v3 is a 0.6B model that its card describes as a multilingual automatic speech recognition model for high-throughput transcription, with a FastConformer encoder and a TDT decoder, automatic punctuation and capitalisation, and word-level and segment-level timestamps. It covers 25 European languages and detects the language itself without prompting. It expects 16 kHz mono audio in .wav or .flac. The card reports handling audio up to 24 minutes with full attention on an A100 80GB, or up to three hours with local attention; treat those as the publisher’s figures for their hardware, not yours.

!Important

Parakeet’s licence is CC-BY-4.0, not Apache-2.0 like most of the models in this course. That is a permissive licence with an attribution condition, which is a real obligation if you put transcription into a product. The model reference records the licence for every model the course uses; read it there before you build on one.

It runs through NVIDIA’s NeMo toolkit, whose ASR documentation shows the shape of the call:

Fragment — not complete on its own

import nemo.collections.asr as nemo_asr

asr_model = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v3")
transcript = asr_model.transcribe(["meeting.wav"])[0].text

Whisper-class models are the other family, and they are not in this course’s model table because the course pins Parakeet. They are worth naming because you will meet them everywhere: whisper.cpp describes itself as “high-performance inference of OpenAI’s Whisper automatic speech recognition (ASR) model”, is MIT-licensed, runs on everything from a Raspberry Pi to WebAssembly, and supports integer quantisation of the Whisper weights so a transcription model fits in very little memory. On a machine with no NVIDIA hardware it is often the path of least resistance, and its GGML model files work the same way GGUF files do in Part 6. Check the Whisper weights’ own licence on their model card before shipping anything.

✱Hardware note

Transcription is the one workload in this part where the older, simpler question applies: does the toolkit run here at all? NeMo is a PyTorch stack and follows PyTorch’s platform support, which Part 8 covers per track; whisper.cpp is a C++ project in the llama.cpp mould with Metal, CUDA and CPU backends. If you are on Track X or Track M and NeMo fights you, that is a toolchain problem, not a model problem, and the whisper.cpp path is a legitimate answer rather than a compromise.

Documents: try the text layer first

A PDF is either text with a layout or a picture of text, and the two need entirely different treatment. The mistake is to reach for the vision model first.

Born-digital PDFs carry their text. Extracting it is exact, instant and free of model error. PyMuPDF describes itself as “a high-performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents”, handling PDF, XPS, EPUB, MOBI, FB2, CBZ, SVG, TXT, MD and image formats, with page rendering, text extraction, table extraction, image extraction and OCR integration. Its companion project converts PDFs to Markdown, which is a convenient input for the chunking strategy in the previous lesson because it preserves headings.

!Important

PyMuPDF is dual-licensed: its documentation states that “PyMuPDF and MuPDF are now available under both, open-source AGPL and commercial license agreements”, with Artifex as “the exclusive commercial licensing agent for MuPDF”. The AGPL has obligations that matter if you put it behind a network service, which is exactly what the project in this part does. Read the licence, and choose a differently-licensed extractor if the AGPL does not suit you.

Scans need pixels. Render each page to an image and send it to a vision model, or to a dedicated optical character recognition engine. This is one to two orders of magnitude more expensive per page than reading a text layer, so the sensible pipeline decides per page.

The rule that keeps this cheap:

1. Extract the text layer for every page.
2. Count pages that produced little or no text.
3. Send only those pages through the vision path.
4. Record which pages came from which path, because a reader of the answer deserves to know whether their citation came from a text layer or from a model reading a photograph.

That last point is not bureaucracy. Optical character recognition errors are not random noise; they are plausible substitutions, and a transposed digit in an invoice total is exactly the kind of error that survives every downstream check.

What runs on which track

Track S — NVIDIA DGX Spark

Vision and speech both work. llama.cpp with a projector is the simplest path and runs the same way as on any Linux machine. vLLM’s multimodal support is available here, and Part 9 covers getting vLLM running on this platform. NeMo follows the NVIDIA container path that Part 8 uses for aarch64, which is the smoothest route to Parakeet on this track. 128 GB of unified memory means the resolution and context limits above are advisory rather than binding.

Track X — AMD Ryzen AI Max+ 395Partial

llama.cpp with Vulkan or ROCm covers vision and whisper.cpp covers speech; NeMo and vLLM are the parts to check for your driver stack before relying on them.

llama.cpp’s multimodal path works through the same backend you already built in Part 6, so a vision model is a projector file away. For speech, whisper.cpp is the low-friction option on this track. Parakeet through NeMo depends on your ROCm PyTorch stack, which Part 8 covers with its status dated; treat it as something to verify rather than assume.

Track M — Apple silicon

Two good paths. llama.cpp with Metal handles vision models with a projector, and mlx-vlm is the MLX-native option with a command-line generator, a server and a chat interface. For speech, whisper.cpp has first-class Metal support and is the usual choice here. Unified memory means the image resolution you choose competes directly with everything else open on the machine.

Track N — NVIDIA desktop or laptop

Everything in this lesson runs, and video memory is the binding constraint rather than system memory. On an 8 GB card, keeping the projector off the graphics processor is the lever that makes a vision model fit; on 24 GB and above, the constraint moves to how many images you put in one context. NeMo and vLLM are both at home on this track.

Preserve evidence through modality conversion

A scanned document passes through several transformations before a final answer: page rendering, OCR or vision encoding, extraction and possibly retrieval. Keep page numbers and source identifiers through the pipeline. Otherwise a plausible answer cannot be traced back to the region or timestamp that supports it.

Construct a small evaluation set with a rotated page, a multi-column layout, a table, an illegible field and a clean control. For audio, include silence, overlapping speakers and a proper noun whose spelling matters. Score extraction separately from the final task. A model cannot reliably reason from a total whose digits were transcribed incorrectly.

Define what happens when evidence is unclear: return an uncertainty marker or request review rather than converting unreadable text into a confident value. Record preprocessing resolution, audio segmentation and model identity alongside the answer. These are part of the experiment, since changing them changes both information available to the model and resource use. Text-only token counts do not fully describe the multimodal request’s memory and compute requirements.

Recap

Multimodal input is an encoder and a projector that turn pixels or audio into tokens in the ordinary context, which is why images cost context length, key-value cache and memory in exactly the ways Parts 4 and 6 taught. llama.cpp reaches vision models through libmtmd and a separate projector file, with llama-server exposing them on the OpenAI-compatible chat API; vLLM takes multimodal content through multi_modal_data offline and typed content items online, with a per-prompt limit worth setting; mlx-vlm is the MLX path on a Mac. Qwen3-VL-8B is the course’s reference vision model under Apache-2.0. Parakeet is the pinned speech model, CC-BY-4.0 rather than Apache-2.0, run through NeMo; whisper.cpp is the portable alternative and the easier one on tracks where the PyTorch stack is awkward. For documents, extract the text layer first, count the pages that failed, and send only those through the expensive path, recording which page came from which so that a citation means something.

Check your understanding

Question 1. You send four high-resolution page images in one conversation and the server reports a context overflow. What is the underlying cause?

• The vision encoder ran out of memory
• Images are turned into tokens that occupy ordinary context positions, so four large images can exceed the context length you allocated
• The projector file is the wrong version
• Vision models have a hard limit of one image

Show the answer and why

Answer: Images are turned into tokens that occupy ordinary context positions, so four large images can exceed the context length you allocated

Nothing downstream of the projector knows the tokens came from pictures. The context budget, the key-value cache arithmetic and the overflow behaviour are all the same as for text, which is why the memory lessons in Parts 4 and 6 transfer directly.

Question 2. A colleague sends you a 200-page PDF to make searchable. What should you do first?

• Render every page and send them all to a vision model
• Extract the text layer, count how many pages produced little or no text, and send only those through the vision path
• Fine-tune a model on the document
• Increase the context length so the whole PDF fits

Show the answer and why

Answer: Extract the text layer, count how many pages produced little or no text, and send only those through the vision path

Born-digital pages carry exact text at no model cost. Reserving the expensive, error-prone path for the pages that actually need it is usually one or two orders of magnitude cheaper, and it lets you record which pages were read by a model rather than extracted.

Question 3. Why does the course record that Parakeet is CC-BY-4.0 rather than treating all its models as interchangeable?

• CC-BY-4.0 forbids commercial use
• Because it is a permissive licence with an attribution condition, which is a real obligation once transcription goes into something you ship
• Because it prevents quantisation
• Because it requires a Hugging Face account

Show the answer and why

Answer: Because it is a permissive licence with an attribution condition, which is a real obligation once transcription goes into something you ship

Attribution is a condition, not a formality, and it differs from the Apache-2.0 terms most of the course's models carry. The model reference records the licence for every model precisely so that this is checkable rather than assumed.

Question 4. On an 8 GB NVIDIA card, which lever most directly helps a vision model fit?

• Raising the image resolution so fewer images are needed
• Keeping the multimodal projector off the graphics processor, and lowering the context length
• Using an instruct model instead of a vision model
• Disabling the key-value cache

Show the answer and why

Answer: Keeping the multimodal projector off the graphics processor, and lowering the context length

The projector is extra weights and llama.cpp documents an option to leave it unoffloaded, and the key-value cache scales with context length. Higher resolution moves the wrong way: it produces more visual tokens, which costs both context and cache.

Check my answersClear