Retrieval-Augmented Generation: Embeddings, Chunking and Vector Stores

Fine-tuning a model on your documents is the answer most people reach for and usually the wrong one. Retrieval is the other answer: leave the weights alone, find the relevant few paragraphs at question time, and put them in the prompt. By the end of this lesson you will be able to build that pipeline out of pieces that run entirely on your machine, explain each design decision in it, and describe the four ways it produces an answer that is fluent, sourced and wrong.

Two memories

The idea has a paper. Lewis and colleagues, in 2020, described “a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained parametric and non-parametric memory for language generation”, motivated by the observation that although large pretrained models “store factual knowledge in their parameters”, their “ability to access and precisely manipulate knowledge is still limited”, and that “providing provenance for their decisions and updating their world knowledge remain open research problems”.

Strip the vocabulary and you have the whole design argument. Parametric memory is the weights: enormous, fixed at training time, and impossible to audit or update. Non-parametric memory is an index you own: small, changeable this afternoon, and able to say where an answer came from. Retrieval is the machinery that puts the second in front of the first.

Everything below is engineering on top of that sentence.

Two pipelines, run at different times

1. 1Ingest: readDocuments to plain text, keeping the structure: file, heading path, position.
2. 2Ingest: chunkSplit into passages small enough to embed usefully and large enough to answer from.
3. 3Ingest: embedOne vector per chunk, from a local embedding server.
4. 4Ingest: indexVectors and metadata into a local store you can back up.
5. 5Query: embed the questionSame model, same dimensions, query-side instruction prefix.
6. 6Query: retrieveNearest neighbours, generously: tens of candidates, not five.
7. 7Query: rerankA cross-encoder reads question and candidate together and reorders them.
8. 8Query: answerThe top few chunks in the prompt, with instructions to cite and to refuse.
9. 9Query: verifyCheck that every citation names a chunk that was actually supplied.

The top four steps run once per document change. The bottom five run once per question. Almost every failure in a retrieval system is a mismatch between the two.

Embeddings, and the rules that come with them

Part 2 established what an embedding is: a vector positioned so that texts about similar things sit near each other. A retrieval system is that idea plus bookkeeping.

This course uses Qwen3-Embedding-0.6B, Apache-2.0 licensed, which its card describes as supporting over 100 languages including programming languages, with a context of 32,000 tokens and an output dimension configurable up to 1024 through Matryoshka representation learning, so you can ask for a shorter vector and trade a little quality for a smaller index. The card recommends formatting queries as Instruct: {task_description}\nQuery:{query} and reports that instructions typically improve performance by one to five per cent, and that English instructions should be used even in multilingual settings.

Four rules follow from how embeddings work, and breaking any of them produces a system that retrieves plausible rubbish.

One model, both sides. The vector for the question and the vectors for the chunks must come from the same model at the same dimension. Change the embedding model and the whole index is worthless; rebuild it. Write the model name and dimension into the database when you create it, and refuse to query when they disagree.

Asymmetry is deliberate. The card’s convention puts the instruction on the query and not on the document. That is not an oversight: a question and the passage answering it do not look alike, and the instruction prefix is what tells the model to embed a query as a query. Applying it to both sides quietly changes the geometry.

Similarity is not relevance. Two passages about the same topic sit close together whether or not either answers the question. This is why the reranker below exists.

The index is a copy. Embedding a document does not link to it; it snapshots it. A document edited after ingestion is still in the index in its old form, and nothing will tell you.

RunnableAll tracks

an embedding-only server on its own port

llama-server \
  --model ~/models/qwen3-embedding-0.6b/Qwen3-Embedding-0.6B-Q8_0.gguf \
  --alias qwen3-embedding \
  --embedding \
  --ctx-size 8192 \
  --host 127.0.0.1 \
  --port 8090

The README documents --embedding, --embeddings as restricting the server to “only support embedding use case”, with both an OpenAI-compatible /v1/embeddings endpoint and a native /embedding one. Run it on its own port rather than sharing with the chat model: it is a different model, it is tiny, and you want to be able to restart one without the other.

Chunking is the decision that decides everything else

A chunk is the unit of retrieval and the unit of citation. Choosing its size is choosing what your system can answer.

Too small and each chunk embeds cleanly but lacks the context to answer from: you retrieve the sentence containing the phrase and lose the paragraph that qualified it. Too large and the chunk’s vector is an average of several topics, so it matches everything weakly and nothing strongly, and you waste context sending three pages to answer one question.

The strategies worth knowing, in the order you should try them:

1. Structural. Split on the document’s own boundaries: headings, sections, list items, table rows. Documents that have structure should be chunked by it, because the author already grouped related sentences for you.
2. Fixed window with overlap. Pack text into windows of a few hundred tokens with an overlap of ten to twenty per cent, so a fact that lands on a boundary appears whole in one of the two neighbouring chunks. This is the default when there is no structure.
3. Sentence window. Embed single sentences for precise matching but return the surrounding paragraph for the answer. More machinery, better results on dense reference text.
4. Parent document. Embed small, return the whole section the small chunk came from. The same trade in a coarser form.

The project in this part uses structural splitting on headings, then packs paragraphs into windows with overlap, and prepends the heading path to each chunk’s text. That last detail matters more than it looks: a chunk that begins “The window is 02:00 to 04:00 on Sundays” matches almost nothing, and the same chunk beginning “Service runbook › Backups › The window is 02:00 to 04:00 on Sundays” matches a question about backups.

✕Common mistake

Chunking by character count with no overlap is the fastest way to build a system that fails on precisely the questions you care about, because the interesting facts are short and land on boundaries. If you take one thing from this section: overlap, and keep the heading path.

Store metadata with every chunk: source path, heading path, chunk index, and a hash of the source file. The first three are your citations. The fourth is how you detect that a document changed and its chunks need replacing.

A vector store that is a file

You can run a vector database. On one machine, for a few thousand documents, you should not have to.

This course uses sqlite-vec, which its documentation describes as “a vector search SQLite extension that runs anywhere”, dual-licensed MIT and Apache-2. The reasons are the ones that matter for a course about owning your own stack: the entire index is one SQLite file you can copy, back up and delete; there is no server to run, secure or keep alive; the query language is SQL you already know; and the chunks and their vectors live in the same file, so a citation is a join rather than a second lookup in a second system.

The whole retrieval store

1. chunks tableOrdinary SQLite: chunk id, source path, heading path, text, source hash.your citations
2. vec0 virtual tableThe vectors, one row per chunk, joined back by id.the search
3. meta tableEmbedding model name, dimension, ingestion date. Checked on every query.the safety catch
4. One .db fileCopy it, back it up, ship it to another machine. There is no other state.

No daemon, no port, no second thing to secure. The lesson on serving beyond localhost has enough to worry about already.

A vector table is created as a virtual table, with the dimension fixed at creation and the distance metric chosen there too:

Pseudocode — not a real command

create virtual table vec_chunks using vec0(
  chunk_id integer primary key,
  embedding float[1024] distance_metric=cosine
);

and a nearest-neighbour search is a match with a k, joined back to the ordinary table that holds the text:

Pseudocode — not a real command

with knn as (
  select chunk_id, distance from vec_chunks
  where embedding match :query_vector and k = 50
)
select chunks.source, chunks.heading, chunks.text, knn.distance
from knn left join chunks on chunks.id = knn.chunk_id
order by knn.distance;

From Python the extension is loaded onto an ordinary connection, and vectors are passed as compact blobs:

Fragment — not complete on its own

import sqlite3, sqlite_vec
from sqlite_vec import serialize_float32

db = sqlite3.connect("qa-index.db")
db.enable_load_extension(True)
sqlite_vec.load(db)
db.enable_load_extension(False)
# ... then pass serialize_float32(vector) as the query parameter

⌚Version note

sqlite-vec is pre-1.0: the project page shows a 0.1.x alpha version and states that its documentation is a work in progress, and it needs SQLite 3.41 or later. That is an acceptable risk for an index you can rebuild from your documents in minutes, and it would not be for your only copy of the documents themselves. Checked on 2026-09-08. LanceDB and Chroma are the two other local stores you will see recommended; this course does not use them and makes no claim about them either way.

Reranking: the step most tutorials skip

Vector search is a cheap approximation. It compares two vectors that were computed independently, so nothing in the comparison ever looked at the question and the passage at the same time. A reranker does exactly that, and it is much better at it.

Qwen3-Reranker-0.6B, Apache-2.0 and multilingual, is prompted with an instruction, a query and a document, and judges “whether the Document meets the requirements based on the Query”, scoring from the logits of the “yes” and “no” tokens. Its card documents a default instruction of “Given a web search query, retrieve relevant passages that answer the query”, and reports its own evaluation on “the top-100 candidates retrieved by dense embedding model Qwen3-Embedding-0.6B”, which is the pairing this course uses at a smaller candidate count. llama-server exposes it: --rerank, --reranking “enable reranking endpoint on server”, with /rerank, /v1/rerank and /v1/reranking as aliases.

The shape of the pipeline follows from the cost. Retrieval over ten thousand chunks is a vector comparison per chunk and is effectively free. Reranking is a forward pass over the query and the candidate together, so its cost is candidates multiplied by chunk length, in prefill. Retrieve fifty candidates cheaply, rerank them, keep five. Retrieving five and reranking them changes nothing, because the mistake you are trying to fix was made when the right chunk failed to reach the top five.

▤Memory budget

A reranker over fifty candidates of four hundred tokens each is twenty thousand tokens of prefill for one question. On a small machine that is a real cost, and it is a compute cost rather than a memory one, so it shows up as latency and not as a failure to load. Halve the candidate count before you drop the reranker: it is usually the highest-value component in the whole pipeline.

Citations, grounding and the refusal

An answer without a citation cannot be checked, and a system that cannot say “not in these documents” will invent something instead.

Both are prompt-and-validation problems, which is where the previous lesson pays off. Give every chunk you put in the prompt a short identifier. Require the answer to be an object with a list of claims, each carrying the identifiers it rests on, plus a boolean saying whether the documents supported an answer at all. Then verify in code: every identifier the model returned must be one you supplied. A citation to a chunk that was not in the prompt is a fabrication and should be treated as a failed answer, not shown to a reader with a footnote.

The refusal path needs the same care. “Answer only from the documents below; if they do not contain the answer, say so” is a start, and it will be ignored some of the time. Making the refusal a field in a schema, rather than a sentence the model must remember to produce, moves it from a request to a structural option. Then measure how often it is used correctly, with questions you know the corpus cannot answer. The project in this part includes those questions for exactly this reason.

The four ways this goes wrong

The right chunk is not retrieved. The answer is in your corpus and the search did not find it. Causes: chunking that split the fact from its context, a query phrased unlike the document, an embedding model weak in that language or domain. Diagnose by searching the index by hand for a phrase you know is in the answer, and see where the correct chunk ranks.

The index is stale. The document changed and the vectors did not. This produces the worst possible failure mode, a correct-looking answer with a real citation to text that no longer exists. Store a hash of each source file and re-ingest what changed; make the ingest script cheap enough to run from a scheduled task.

Confident nonsense from irrelevant context. Retrieval always returns its top k, even when the best match is unrelated. The model is then handed three irrelevant passages and a question, and generally answers anyway. A distance threshold and the reranker’s score are both defences; the refusal path is the last one.

Questions retrieval cannot answer. “How many machines are listed?”, “What changed between these two policies?”, “Summarise everything about backups.” These need every chunk, or a comparison across chunks, and nearest-neighbour search over a top-k will not deliver it. Know this class exists so that you recognise it in an evaluation instead of blaming the model.

When a fine-tune is the better tool

The division is clean enough to state as a rule.

Retrieval changes what the model reads. Use it for facts: things that are true today, change next month, are specific to you, and need to be cited. Adding a document is a file copy and an ingest run.

Fine-tuning changes how the model behaves. Use it for form: a house output format, a domain vocabulary, a tone, a task shape the model keeps getting slightly wrong. Part 13 covers it, and the evaluation set from this part’s lab is how you will tell whether it helped.

Attempting the first with the second is the classic waste: a fine-tune on a thousand company documents produces a model that writes in your house style and still cannot tell you what is in document 634. Attempting the second with the first is subtler but common: no quantity of retrieved examples reliably fixes a model that will not follow your output format, which is what constrained decoding and fine-tuning are for.

The honest answer for a mature system is usually both, in that order: retrieval first, because it is cheaper and reversible, and a fine-tune afterwards for the behaviour that retrieval never fixed.

Evaluate retrieval before evaluating the generated answer

Create a question table with the authoritative document and passage that should support each answer. Include questions whose answer is absent. Run retrieval alone and inspect whether the supporting passage appears in the candidates and in the final context. This isolates extraction, chunking, indexing and ranking from generation.

For an incorrect answer, walk backwards: did the source contain the fact, was it extracted intact, was the right chunk retrieved, and did the generator preserve it? A table split across chunks may require different extraction or chunking; a correct passage ranked too low may require retrieval changes. A fluent answer that contradicts supplied evidence is a generation or instruction-following failure.

Document changes also need a lifecycle. Keep source identifiers, revisions and access permissions with chunks. Remove obsolete or forbidden chunks from retrieval and rebuild embeddings when the embedding contract changes. Evaluate stale-answer cases after an update. A RAG system is current only if its ingestion and deletion paths maintain the index; adding retrieval once does not keep facts fresh automatically.

Recap

Retrieval gives a model a second, non-parametric memory that you own and can update. The pipeline is ingest — read, chunk, embed, index — and query — embed, retrieve, rerank, answer, verify. Chunking is the decision that determines what the system can answer: split on the document’s own structure, overlap, and keep the heading path in the text. Query and document must share an embedding model and dimension, and the query-side instruction prefix belongs on the query only. sqlite-vec keeps the vectors and the text in one file with no server, which is the right shape for one machine. A reranker reading query and candidate together fixes the ordering that independent vectors got wrong, so retrieve widely and rerank down. Citations must be verified in code, and the refusal path must be a field, not a hope. Four failure modes recur: the chunk that was not retrieved, the stale index, confident answers from irrelevant context, and questions no top-k search can serve. Retrieval is for facts; fine-tuning is for behaviour.

Check your understanding

Question 1. You improve your embedding model and swap it into an existing pipeline without rebuilding the index. What happens?

• Retrieval quality improves immediately
• Nothing changes until the next ingest
• Retrieval degrades badly, because query vectors from the new model are being compared with chunk vectors from the old one
• The store raises a dimension error and refuses the query

Show the answer and why

Answer: Retrieval degrades badly, because query vectors from the new model are being compared with chunk vectors from the old one

Distances are only meaningful between vectors from the same model. If the dimension happens to match, nothing errors and the system silently returns nonsense, which is why the course writes the model name and dimension into the database and checks them on every query.

Question 2. Why does the pipeline retrieve fifty candidates and rerank them, rather than retrieving the five it will use?

• Because rerankers need at least fifty inputs
• Because the failure being fixed is the correct chunk falling outside the top five on vector similarity alone; a wider net gives the cross-encoder something to rescue
• Because it is faster
• Because fifty chunks fit in the context window

Show the answer and why

Answer: Because the failure being fixed is the correct chunk falling outside the top five on vector similarity alone; a wider net gives the cross-encoder something to rescue

Vector similarity compares two independently computed vectors and is a rough proxy for relevance. The reranker reads the query and the candidate together, so it can promote a chunk that ranked twentieth. It cannot promote a chunk it never saw.

Question 3. Your system answers a question with a citation to a chunk that does not exist. What is the correct response?

• Show the answer without the citation
• Lower the temperature and try again
• Treat the answer as failed: verify in code that every returned identifier was one you supplied, and reject the response when it was not
• Add more chunks to the prompt

Show the answer and why

Answer: Treat the answer as failed: verify in code that every returned identifier was one you supplied, and reject the response when it was not

A fabricated citation is a fabricated answer wearing evidence. The check is cheap and mechanical, and doing it in code rather than by eye is the difference between a system that is grounded and one that looks grounded.

Question 4. Which of these problems is retrieval the wrong tool for? Select all that apply.

• The model must always answer in your organisation's report template
• Staff need answers about policies that change every few weeks
• The model keeps using informal language where the domain requires precise terms
• A question that requires counting every item in the corpus

Show the answer and why

Answer: The model must always answer in your organisation's report template, The model keeps using informal language where the domain requires precise terms, A question that requires counting every item in the corpus

Retrieval changes what the model reads, so it is right for changing facts. Format and vocabulary are behaviour, which is fine-tuning territory. Counting or comparing across the whole corpus needs all the chunks, which a top-k nearest-neighbour search does not provide.

Question 5. What does prepending the heading path to each chunk's text improve?

• The size of the index
• The chunk's vector, because the topic words that the passage itself omits are now present for the embedding model to use
• The speed of the reranker
• The model's output format

Show the answer and why

Answer: The chunk's vector, because the topic words that the passage itself omits are now present for the embedding model to use

An embedding is computed from the chunk text alone. A passage that says "the window is 02:00 on Sundays" contains no word about backups, so it matches no question about backups until its heading path is part of the text being embedded.

Check my answersClear