Agentic Retrieval and Research Agents

By the end of this lesson you will be able to turn Part 10’s retrieval pipeline into a tool an agent decides when to use, add query rewriting and a second hop where they earn their keep, check an answer against the passages it claims to rest on and retrieve again when the check fails, and say with arithmetic how much more all of that costs than asking once.

Retrieval as a tool changes who decides

Part 10 built a pipeline. A question arrives, it is embedded, candidates come back from the index, a reranker reorders them, the top few go into the prompt, and the model answers with citations. The sequence is fixed and your program decides it, which by Part 24’s definition makes it a workflow, and a good one: predictable cost, one model call, and a failure mode you can see.

Agentic retrieval moves one decision from your program to the model: whether to search at all, and what to search for. Everything else follows from that one change.

The research case for it is Self-RAG, whose authors state the problem exactly: “indiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary, or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation”. Their framework “adaptively retrieves passages on-demand, and generates and reflects on retrieved passages and its own generations using special tokens, called reflection tokens”.

Two things about that paper are worth being precise about, because it is routinely over-claimed. It trains a model to emit those reflection tokens; you are not going to do that here, and neither do most systems described as agentic retrieval. What transfers is the shape: retrieve when it helps, judge what came back, and be willing to say the passages do not support an answer. You can implement that shape with prompting and with code around the model, and the code part is the part that works.

One turn of a retrieval agent

1. 1The model decides whether to searchIt may answer directly, search, read a specific file, or say the material does not contain an answer. Part 10 had no such decision: it always searched.
2. 2It writes the query itselfThis is query rewriting, and it is where most of the gain is. A question is not a good search string, and the model can turn one into the other.
3. 3Your code runs the same pipeline as Part 10Embed, retrieve candidates, rerank, cap the number of passages. Nothing about this step becomes agentic; it stays a function.
4. 4Passages return as a tool result and stay in the transcriptThis is where the cost is. Every later turn re-reads them unless you take them out.
5. 5The model searches again, or answersA second hop with a query informed by the first result is the capability that a pipeline cannot express.
6. 6A check runs on the answer, outside the modelCitations must name passages that were actually supplied. A failed check is a reason to retrieve again, not a reason to print a warning.

The only steps the model owns are the decisions. The retrieval itself is the same code Part 10 wrote, called from a tool.

Query rewriting is where the gain is

A user’s question and the passage that answers it often share very few words. Part 10 handled part of this with an instruction prefix on the query embedding, which tells the embedding model that it is embedding a question rather than a passage. Agentic retrieval adds the other half: the model writes several search strings and tries them.

Three rewrites are worth having as habits, and all three are things a small model does well:

• Terms rather than a question. “When is the backup window?” becomes backup window schedule Sunday. Vector search tolerates the question form; keyword and hybrid search does not.
• Entity expansion. A question about “the spare machine” searches better once the model has learnt from a first result that the spare machine is called skua. This is the cheapest form of multi-hop and often the only one needed.
• Decomposition. “Which machine serves inference if the main one fails, and why can the spare not do it?” is two searches. A model that issues one search for both halves gets passages that half-answer each.

✦Tip

Cap the searches. A model with a search tool and no limit will search four times with four paraphrases of the same query and put four overlapping passage sets in its transcript. In Part 24’s loop the repeat detector catches identical calls; near-identical ones need a different guard, and the simplest is a hard cap on searches per task plus a de-duplication of passages you have already returned.

Multi-hop, and knowing when to stop

Multi-hop means the second query depends on the first result. It is the capability that justifies the whole pattern, and it is also where small models wander.

The pattern that works on local models is narrow: hop only when the first result names something the question did not. A retrieved passage that mentions a machine, a file, a policy or a date the question did not contain is a lead; a retrieved passage that merely restates the question is not. That rule can live in the system prompt, and unlike most prompt rules it is checkable afterwards from the trajectory, because you can see the queries.

Everything else about stopping is your program’s job, as it was in Part 24. A maximum number of searches, a maximum number of turns, and a de-duplicated passage set are three lines of code that do more for multi-hop reliability than any prompt.

Source routing: not everything lives in one index

Once retrieval is a tool, having several is natural, and immediately raises the question of which one to call. This is Anthropic’s routing pattern again, applied to sources instead of tasks, with the same condition: distinct categories that are better handled separately.

Sources, in the order an agent should prefer them

cost and trust both fall as you go down

1. The transcriptSomething already retrieved this turn. Free, and the most-missed opportunity: agents re-search for facts already in their own context.free
2. A scratchpad or notes fileWhat this agent wrote down earlier in the task. Cheap to read, under your control.cheap
3. Your document indexPart 10's index over documents you chose. Known provenance, known freshness, and the answers the task suite checks.
4. The filesystem in the workspaceFiles the index does not cover, read by name. Still yours; still trusted as data rather than as instructions.
5. A self-hosted web searchPublic results, unknown provenance, arbitrary content. Useful, and the point at which everything the fifth lesson is about begins to apply.untrusted

A routing rule as simple as 'index first, web only if the index returned nothing' removes most of the risk and most of the cost of giving an agent the internet.

Give each source a separate tool with a description that says what is in it. Part 24 made the argument: a model choosing between search_documents and search_web is making a decision the descriptions can make obvious, while a model choosing between search, find and lookup is guessing.

Faithfulness checks, and re-retrieval as the response

An agent that retrieves and then answers from memory anyway is the failure this section exists to catch, and it is common because it is invisible: the answer is fluent, the citations look plausible, and nothing errors.

Part 10 built the check that catches it, and it belongs here unchanged. The answer arrives as a structured object with a citation list; every citation must name a passage that was actually supplied; an answer citing anything else is not shown as sourced. That check is deterministic, costs nothing, and cannot be persuaded.

What is new here is what you do when it fails. In a pipeline, a failed check is an error message. In an agent, it is an observation, and the loop can respond to it:

A failed faithfulness check as a retrieval trigger

1. 1The answer cites a passage identifier that was never suppliedDetected by set membership, in your code, before anything is shown to anyone.
2. 2The failure goes back as a tool result"Citation C7 was not among the passages supplied. Valid identifiers were C1 to C5." A specific, actionable observation.
3. 3The model searches again or narrows its claimUsually one of two things happens: it finds the passage it was half-remembering, or it drops the unsupported sentence.
4. 4On a second failure, stop and report unanswerableTwo rounds is the whole benefit. Beyond that you are paying for the model to keep guessing.

The same check that rejects an answer in a pipeline steers the loop in an agent. The check is the valuable part; the loop is what makes it recoverable.

Local web search, and its limits

If an agent is going to search the web from a machine you own, the self-hosted option is SearXNG, which its documentation describes as “a free internet metasearch engine which aggregates results from up to 269 search services” where “users are neither tracked nor profiled”. The documentation read for this lesson showed version 2026.9.8+3fdc6d753.

The part that matters for an agent is the search API. It answers on /search for both GET and POST, and takes q for the query, plus optional categories, language, pageno, time_range (day, month or year), safesearch and format. JSON is one of the output formats, alongside CSV and RSS, and the documentation is explicit that a format “needs to be activated in search: settings” in the instance’s configuration file, or the request is refused with a 403.

RunnableAll tracks

one JSON query against a SearXNG instance you run

curl --silent --get \
  --data-urlencode "q=qwen3 embedding instruction prefix" \
  --data-urlencode "format=json" \
  --data-urlencode "language=en" \
  http://127.0.0.1:8888/search | head -c 400

Now the limits, which are more interesting than the capability.

It is a metasearch front end, not a search engine. Results come from upstream services, so quality, availability and rate limiting are theirs. An instance that worked yesterday can return nothing today because an upstream started refusing it, and your agent will experience that as a task it cannot do rather than as an error.

Snippets are not documents. The API returns titles, URLs and short extracts. Answering from a snippet is answering from an advertisement for a page. Actually reading the page means a second tool that fetches it, and that tool is a different risk entirely.

Fetching a page is accepting arbitrary text into your context. OWASP’s prompt-injection entry defines indirect injections as occurring “when the model processes external content (websites, files) containing data that alters behavior when interpreted by the model”. A web fetch tool is that channel, deliberately opened. The fifth lesson is about what to do; the minimum, here, is that page text enters as a tool result marked as data, never as a system or user message, and that the agent holding it should not also hold a tool that writes anywhere.

⛨Security

Giving one agent private documents, a web fetch and any outward channel is the combination Simon Willison named the lethal trifecta, and the fifth lesson takes it apart properly. If you add web search to a research agent this week, split it: one agent searches the web and returns text, a different agent holds the private index, and neither has a tool that can send anything anywhere.

What it costs, against Part 10’s pipeline

Part 10’s ask.py is the baseline: one embed, one retrieval, one rerank, one generation. Its cost is a single call.

Pseudocode — not a real command

k  = passages put in the prompt
p  = tokens per passage
S  = system prompt + tool schemas
Q  = the question
O  = tokens generated

Plain pipeline (Part 10 ask.py)
    input  = S + Q + k * p          read once
    output = O
    model calls = 1

Agentic retrieval, r searches over n turns
    input  ≈ n * (S + Q)
             + k * p * (sum over searches of the turns each result survives)
           ≈ n * (S + Q) + k * p * r * n / 2
    output ≈ n * O
    model calls = n

Ratio of passage tokens read ≈ r * n / 2

That last line is the one to remember. Three searches over eight turns re-read the retrieved text on the order of twelve times more than the pipeline reads it once. On a hosted model that is a bill. On a 30B mixture-of-experts model on your own machine it is prefill time, repeated, on the machine you are waiting for.

Which gives a clear rule about when the extra cost is justified:

Situation	Use
One question, answer is in one passage	Part 10’s pipeline. The agent adds nothing but variance.
The right query is not the user’s words	Agentic, for the rewriting alone; often two turns is the whole benefit.
The answer needs a fact found by a previous search	Agentic. A pipeline cannot express the second hop.
Several sources with different content	Agentic, with one tool per source and a routing rule.
High volume, fixed question shape	Pipeline, and spend the savings on a better reranker.

Three mitigations bring the agentic cost down a long way, and all three are code rather than prompting:

• Cap k hard. Six passages is generous. The reranker exists so that six good ones beat twenty mixed ones.
• Replace passages with notes. After the model has used a passage set, swap it in the transcript for a two-line summary of what it contained. This is Part 24’s compaction, applied to the biggest thing in the transcript.
• Write findings to a scratchpad. A file the agent writes and re-reads by name keeps retrieved text out of the context entirely until it is needed again.

Require evidence at each research hop

A research agent can rewrite a query, select a source, retrieve a passage and decide to search again. Each step should preserve the question it is trying to answer and the evidence obtained. Otherwise later summaries can turn an earlier guess into an apparently established fact.

Create a task requiring two linked facts from different documents. Record the retrieved passage identifiers at each hop and check that the final conclusion follows from them. Include an unanswerable version of the task and verify that the agent stops with the missing evidence stated rather than inventing another source.

Compare against the fixed retrieval pipeline from Part 10. Count searches, model calls, tokens and latency along with answer correctness and citation support. Query rewriting earns its cost when it retrieves evidence the baseline misses. Additional searches that repeat the same passages add cost without new information. Set a stopping rule based on evidence coverage, duplicate retrieval and budget, and treat instructions embedded in retrieved sources as untrusted content.

Recap

Making retrieval a tool moves exactly one decision to the model: whether to search and what to search for. The gain is mostly query rewriting, with genuine multi-hop as the capability a pipeline cannot express, and the rule that keeps hops sane is to hop only when a result names something the question did not. Several sources want several tools with descriptions that make the choice obvious, and a routing rule that prefers the transcript, then a scratchpad, then your index, then the filesystem, and only then the web. Part 10’s citation check belongs here unchanged, except that a failure becomes an observation the loop can act on rather than an error, for at most two rounds. Self-hosted web search through SearXNG is available and documented, needs its JSON output enabled in the instance settings, returns snippets rather than documents, and opens the indirect-injection channel deliberately. And the cost, against Part 10’s pipeline, is roughly r * n / 2 times the passage reading, which is why capping the passage count, compacting used passages and writing findings to a scratchpad are not optimisations but part of the design.

Check your understanding

Question 1. Your agent performs three searches over eight turns, each returning five passages. Compared with Part 10's single-shot pipeline, roughly how much more retrieved text does the model read?

• About three times, once per search
• About twelve times, because each result stays in the transcript and is re-read on every later turn
• The same, because the passages are the same
• About eight times, once per turn

Show the answer and why

Answer: About twelve times, because each result stays in the transcript and is re-read on every later turn

Passages retrieved on an early turn are resent on every subsequent turn. With r searches over n turns the passage reading is on the order of r * n / 2 times the single read, which for r = 3 and n = 8 is about twelve. This is why compacting used passage sets matters more than any other optimisation in a retrieval agent.

Question 2. Which of these is the strongest reason to make retrieval agentic rather than a pipeline?

• Agents are more capable than pipelines
• The second query depends on what the first one found, which a fixed sequence cannot express
• It uses fewer tokens
• It removes the need for a reranker

Show the answer and why

Answer: The second query depends on what the first one found, which a fixed sequence cannot express

Multi-hop is the capability difference. Query rewriting is the common gain and can often be had in two turns; everything else a pipeline does at lower cost and with less variance. Agentic retrieval uses considerably more tokens, not fewer, and the reranker is just as necessary.

Question 3. A SearXNG instance you run returns 403 for a JSON query although the same query works in a browser. What is the most likely cause?

• The instance is rate-limited by an upstream engine
• The JSON output format has not been activated in the instance settings, which the documentation says is required
• The query needs a POST rather than a GET
• The language parameter is missing

Show the answer and why

Answer: The JSON output format has not been activated in the instance settings, which the documentation says is required

The search API documentation states that a requested output format must be enabled in the instance configuration or the request is refused. Upstream rate limiting is real and worth expecting, but it shows up as empty or partial results rather than as a refusal from your own instance.

Question 4. Your agent's answer cites passage C7, but only C1 to C5 were supplied. What should the system do?

• Show the answer with a warning that one citation could not be verified
• Send the failure back as an observation naming the valid identifiers, let the agent search again or narrow its claim, and stop after a second failure
• Silently drop the citation and show the answer
• Increase the number of passages retrieved

Show the answer and why

Answer: Send the failure back as an observation naming the valid identifiers, let the agent search again or narrow its claim, and stop after a second failure

The deterministic check is the same one Part 10 built. What agentic retrieval adds is recovery: a specific observation is something the loop can act on, and often the model either finds the passage it was half-remembering or drops the unsupported sentence. Two rounds is the whole benefit; after that it is guessing at your expense.

Check my answersClear