Local Coding Assistants: Autocomplete and Chat in Your Editor

An editor assistant running on your own hardware is the most convincing demonstration of local inference there is, and the one most likely to disappoint, because it is the one use where latency is not a comfort but a correctness condition. By the end of this lesson you will be able to set up completion and chat against local servers, work out from your own machine’s numbers whether the completion will arrive in time to be useful, choose a model that fits that budget, and demonstrate from the server’s log that nothing you typed left the machine.

Two jobs, two models, two budgets

“AI in the editor” is two different features wearing one name.

Completion fires as you type. It sees the code before and after your cursor, and it produces a few lines. You did not ask for it, so it has to be finished before you have typed past the place it was suggesting. Its budget is a fraction of a second.

Chat, edit and apply fire when you ask. You have stopped typing and you are waiting. The budget is seconds, and a bigger and slower model is worth it.

Running one model for both is the commonest configuration mistake in a local setup. A 30B-class coder gives excellent chat and completions that arrive after you have moved on; a 1.5B base model gives snappy completions and chat you would not use twice.

Fill in the middle

Completion is not the same task as continuation. The model has code on both sides of the cursor, and using the code after the cursor is what makes the suggestion fit. That capability is called fill in the middle, and it comes from a specific pretraining objective: the model was shown documents cut into a prefix, a suffix and a middle, and asked to produce the middle.

The wiring is a set of special tokens. The Qwen2.5-Coder repository documents the file-level form as:

Pseudocode — not a real command

prompt = '<|fim_prefix|>' + prefix_code + '<|fim_suffix|>' + suffix_code + '<|fim_middle|>'

The prefix comes first, then the suffix, then the marker that says “now write the middle”. The repository also notes that “FIM is supported in every version of Qwen3-Coder”, so the capability is not restricted to the older generation. Some coder families add repository-level tokens so that other files can be supplied as context; llama.cpp’s server refers to these as FIM_REPO and FIM_FILE_SEP and states that “if the model has FIM_REPO and FIM_FILE_SEP tokens, the repo-level pattern is used”.

You do not normally build that string yourself. llama-server exposes an /infill endpoint that takes input_prefix, input_suffix and an input_extra array of filename-and-text objects, and assembles the model’s own token pattern from the GGUF metadata. That is the endpoint an editor plugin calls.

One keystroke, one completion

1. 1You stop typingThe plugin waits out a short debounce so it is not firing on every character.
2. 2The plugin builds a requestText before the cursor, text after it, and optionally chunks from other open files.
3. 3POST /infillThe server wraps prefix and suffix in the model's own FIM tokens.
4. 4PrefillThe server reads the context. Cached from the previous keystroke, mostly, if the plugin keeps the prefix stable.
5. 5DecodeA few dozen tokens, one at a time. This is the part your decode rate governs.
6. 6Ghost textShown inline, or discarded because you already typed something else.

Everything above the dashed line is the editor's job and everything below it is the server's. The two places latency comes from are the prefill of a long prefix and the decode of the suggestion.

✕Common mistake

Pointing a completion plugin at an instruction-tuned chat model is the classic error. You get a polite paragraph, a code fence, and an explanation of what the function does, inserted into your source file. Completion wants a base or coder model with FIM tokens; chat wants the instruct model. They are different files.

The latency arithmetic

Do this before you install anything, because it decides which model you can use.

A completion is two costs. The prefill cost is the number of context tokens divided by your machine’s prompt-processing rate, and it is close to zero when the prefix has not changed since the last keystroke and the server’s prompt cache is doing its job. The decode cost is the number of tokens in the suggestion divided by your decode rate, and it is not avoidable.

You measured both rates in Part 5’s lab and again in Part 6’s benchmark, for models of several sizes on your own track. Take the decode rate you recorded for the smallest model you would consider and divide the length of a typical suggestion by it. A three-line suggestion is roughly thirty to sixty tokens. If that division lands beyond a couple of hundred milliseconds you are building something that will annoy you, and the answer is a smaller model rather than a faster computer.

Pending validationWhere the budget goes for one inline completion (estimates from arithmetic, not measurements)

Stage	Typical budget, ms	What sets it
Debounce in the editor	150	A plugin setting. Too short wastes work; too long feels dead.
Prefill, prefix unchanged	20	Cache lookup only. This is why prompt caching matters here.
Prefill, prefix changed	200	Context tokens divided by your prompt-processing rate from Part 5.
Decode, 40-token suggestion	400	40 divided by your decode rate for the completion model.
Round trip and rendering	20	Loopback HTTP. Larger if the server is on another machine.

any track; the row values are worked arithmetic, not measurements · llama.cpp v0.4.0 · 1.5B to 3B-class coder, fill-in-the-middle, Q4_K_M · 4,096 tokens of context · 2026-09-08

Replace every row with your own arithmetic: take the decode and prompt-processing rates you recorded in Part 5 for your track and divide. The validation pass will publish measured values here.

Two consequences follow from that table. The first is that prompt caching is the feature that makes editor completion viable, not an optimisation on top of it. The llama.vim README describes exactly this design goal, saying it “supports very large contexts even on low-end hardware via smart context reuse” through a ring buffer of chunks from open files and yanked text. The second is that the suggestion length is a latency setting. Capping the completion at a few dozen tokens is not stinginess; it is the difference between a feature and an irritation.

Choosing the completion model

Size first, quality second, because a good suggestion that arrives late is not a suggestion.

The llama.vim README publishes a recommendation by available video memory: a 30B-class model above 64 GB, a 7B-class model above 16 GB, a 3B-class model below 16 GB and a 1.5B-class model below 8 GB. Those are the maintainers’ recommendations for their own plugin rather than a measurement of yours, and they line up with the arithmetic above: the machines with less memory are also the machines with less bandwidth, and both point at the smaller model.

For this course, on the four tracks:

• Chat, edit and apply. Qwen3-Coder-30B-A3B-Instruct where memory allows, which its card puts under Apache-2.0 with a native context of 262,144 tokens and only 3.3B active parameters per token, so it decodes far faster than its 30B total suggests. The model reference records it at the 24 GB tier. Below that, Qwen3-4B, or the smaller Qwen2.5-Coder and Qwen3-Coder siblings, which are not in this course’s reference table and should be checked on their own model cards for licence and context before you rely on them.
• Completion. A base or coder model in the 1.5B to 3B class with FIM tokens. The course does not pin one, because the right answer is whichever of them your own arithmetic says fits the budget on your track, and because you can now measure that.

✱Hardware note

On Track M the completion model and the chat model both come out of the same unified memory pool as everything else you are running, so two servers plus an editor plus a browser is a real budget question rather than a theoretical one. On Track N the constraint is video memory and the two models compete for it directly; running the completion model on the graphics processor and the chat model on the processor is a legitimate split when memory is tight, at the cost of chat becoming noticeably slower.

The three integrations worth knowing

llama.vscode is maintained by the same organisation as llama.cpp and described in its README as a “local LLM-assisted text completion, chat with AI and agentic coding extension for VS Code”. It expects a llama.cpp server and is MIT-licensed.

llama.vim is the Vim and Neovim equivalent, described simply as “local LLM-assisted text completion”, also MIT-licensed. It is configured with two endpoints, one for fill-in-the-middle completion and one for instruction-following, which is the two-model split from the top of this lesson made explicit in the configuration.

Continue is the cross-editor option, “an open source AI code assistant for VS Code and JetBrains”, with modes for agent, chat, edit and autocomplete. It is configured with a config.yaml whose models list gives each model a name, a provider, a model, an optional apiBase, and a list of roles drawn from chat, autocomplete, embed, rerank, edit, apply and summarize. Its llama.cpp provider page documents provider: llama.cpp with an apiBase such as http://localhost:8080.

RunnableAll tracks

continue-config.yaml

# Purpose: point the Continue extension at local servers only - a chat and edit model on one
#          port, a fill-in-the-middle completion model on another - so that nothing typed in
#          the editor leaves the machine. Two servers rather than one because chat and
#          completion want different models and very different latency budgets.
# Platform: all (VS Code and JetBrains; the servers may be on this machine or another on
#           your own network)
# Minimum memory: 8 GB for the completion model alone; 24 GB or more to run a
#                 30B-class coder for chat at the same time. See the page for the per-track
#                 choice and for what to run instead on a smaller machine.
# Assumes: Continue installed in the editor, and this file saved as ~/.continue/config.yaml
#          (Continue reads YAML configuration from its own directory; check the extension's
#          documentation for the path on your platform if it does not pick this up).
#          Two llama-server instances are running:
#            port 8080  chat and edit model, started with --jinja
#            port 8081  completion model, a base model with FIM tokens
#          Replace both apiBase values with your gateway address to route through Part 9.

name: local-coding-assistant
version: 0.0.1
schema: v1

models:
  # Chat, edit and apply. Latency here is measured in seconds and that is acceptable,
  # because you asked a question and are waiting for an answer.
  - name: Local chat
    provider: llama.cpp
    model: qwen3-coder-30b-a3b
    apiBase: http://127.0.0.1:8080
    roles:
      - chat
      - edit
      - apply
    defaultCompletionOptions:
      temperature: 0.7
      maxTokens: 2048

  # Autocomplete. Latency here is measured in the low hundreds of milliseconds and anything
  # slower is worse than nothing, because the suggestion arrives after you have typed past
  # it. Use the smallest model that produces useful completions on your hardware.
  - name: Local autocomplete
    provider: llama.cpp
    model: local-fim-model
    apiBase: http://127.0.0.1:8081
    roles:
      - autocomplete
    defaultCompletionOptions:
      temperature: 0.2
      maxTokens: 256

# Alternative provider blocks, for the same two roles.
#
#   provider: ollama       with apiBase http://127.0.0.1:11434 and model set to the Ollama
#                          tag, for example qwen3:4b
#   provider: lmstudio     with apiBase http://127.0.0.1:1234
#   provider: openai       with apiBase set to any OpenAI-compatible base URL, which is how
#                          you point Continue at the Part 9 gateway and get one key, one
#                          address and a usage log for editor traffic as well
#
# Every one of these keeps the request on your own network. The privacy claim is only worth
# making if you check it: the lesson page has you read the server log while you type.

context:
  - provider: file

rules:
  - Answer with code first and prose second.
  - When you are not sure a symbol exists in this repository, say so instead of inventing it.

Download continue-config.yaml67 lines

That file assigns the roles to two different servers, which is the arrangement the latency arithmetic argues for. Point both apiBase values at the Part 9 gateway instead and you get one address, one key and a usage log covering editor traffic as well, at the cost of one more process in the path.

RunnableAll tracks

the completion server: small model, short answers, loopback only

llama-server \
  --model ~/models/coder-fim/coder-1.5b-Q4_K_M.gguf \
  --alias local-fim-model \
  --ctx-size 4096 \
  --predict 64 \
  --parallel 2 \
  --host 127.0.0.1 \
  --port 8081

Proving that nothing leaves

This is the reason many people run a local assistant at all, so it deserves better than an assumption.

The claim is precise: the model traffic goes to a process on your own machine and nowhere else. Three checks establish it.

First, watch the requests arrive. Start the completion server with verbose logging, put the terminal beside the editor, and type. Every suggestion should be preceded by a request in that log, and the log is on your machine.

RunnableAll tracks

watch the requests as you type

llama-server \
  --model ~/models/coder-fim/coder-1.5b-Q4_K_M.gguf \
  --alias local-fim-model \
  --host 127.0.0.1 \
  --port 8081 \
  --verbose

Second, confirm what the server is bound to. Part 7 established the habit and the command: a completion server has no reason to listen anywhere but the loopback address.

RunnableAll tracks

the completion server should answer only to this machine

ss -ltnp | grep -E ':(8080|8081) '

Output — what you should see

LISTEN 0  4096  127.0.0.1:8080  0.0.0.0:*  users:(("llama-server",...))
LISTEN 0  4096  127.0.0.1:8081  0.0.0.0:*  users:(("llama-server",...))

Third, and least obvious: the model traffic being local says nothing about the extension. An editor plugin is an ordinary program that can have telemetry, an update check and an account system of its own, and those are separate from where the tokens go. Read the extension’s own settings, turn off what you do not want, and treat the two questions separately rather than concluding that a local model makes the whole editor offline.

⛨Security

A completion model sees the file you have open, which over a working day means it sees everything: credentials in a configuration file you were editing, a customer’s data in a test fixture, an unreleased design. That is fine when the server is on your own machine. It is a decision that needs making deliberately when the server is on somebody else’s, and it is the reason the next lesson exists. If you put a completion server on your home network for several people to share, it now has a copy of everybody’s keystrokes in its logs by default, and you should either turn that logging off or say so out loud.

What local completion will not do

It will not do a cross-file refactor, it will not run your tests, and it will not chase a failure across five files. Those are agent tasks, they need tool calling and a loop, and they are Part 24 and Part 25. Keep the expectation where the evidence is: a local model in the editor is very good at finishing the line you were already writing, and at answering a question about code you can see.

Measure useful completions, not generated characters

An editor suggestion has value only if it arrives while the user still needs it and preserves the surrounding program. Use a small task set with insertions at a cursor, replacements inside a function and questions about neighbouring files. Keep completion and chat results separate because their contexts and latency budgets differ.

Record the suggestion, its arrival time, whether it was accepted and whether the resulting code passes a relevant check. A suggestion accepted and immediately rewritten is different from one retained. Include cancellation: rapid typing should not accumulate obsolete generation requests behind the latest cursor position.

For privacy verification, inspect every configured provider, including embeddings, reranking and telemetry. A local completion endpoint does not establish where the editor sends indexing or chat requests. Server logs demonstrate requests that arrived locally; they do not prove that no other network destination was contacted. Use the editor’s configuration and network observation together when making that claim. Start on a disposable sample repository before enabling automatic indexing of sensitive projects.

Recap

Editor assistance is two features with different models and different budgets. Completion needs a fill-in-the-middle base or coder model, reached through an endpoint like llama.cpp’s /infill, which assembles the family’s own FIM tokens for you. The budget is arithmetic: suggestion tokens divided by your decode rate, plus a prefill that prompt caching should make nearly free, which is why context reuse is a headline feature of the completion plugins rather than a detail. Chat, edit and apply get the larger model, with Qwen3-Coder-30B-A3B the course’s choice where memory allows. Continue configures both roles in one file against local servers; llama.vscode and llama.vim are the llama.cpp-native options. The privacy property is real and checkable, and checking it means reading the server log and the listening sockets, then asking separately what the extension itself sends.

Check your understanding

Question 1. Why is a fill-in-the-middle model, rather than an instruction-tuned chat model, the right choice for inline completion?

• FIM models are always smaller
• Completion needs the code after the cursor as well as before it, which is what the FIM pretraining objective and its prefix/suffix/middle tokens provide; an instruct model replies conversationally instead
• Chat models cannot produce code
• FIM models do not need a server

Show the answer and why

Answer: Completion needs the code after the cursor as well as before it, which is what the FIM pretraining objective and its prefix/suffix/middle tokens provide; an instruct model replies conversationally instead

The task is different, not just the size. A chat model asked to complete a line returns an explanation and a code fence, which is not what you want inserted at your cursor.

Question 2. Your completion suggestions arrive about a second after you stop typing. Which change is most likely to fix it?

• Raise the temperature
• Increase the context length
• Use a smaller completion model and cap the suggestion length, since decode time is suggestion tokens divided by your decode rate
• Move the server to another machine on the network

Show the answer and why

Answer: Use a smaller completion model and cap the suggestion length, since decode time is suggestion tokens divided by your decode rate

Both terms in the arithmetic point the same way: fewer tokens to decode and a faster decode rate. Moving the server to another machine adds network time to a budget that is already too tight.

Question 3. Why does prompt caching matter more for editor completion than for chat?

• Because completions are longer
• Because consecutive completion requests share almost their entire prefix, so without reuse every keystroke pays full prefill on the whole surrounding file
• Because chat does not use a context window
• Because caching improves suggestion quality

Show the answer and why

Answer: Because consecutive completion requests share almost their entire prefix, so without reuse every keystroke pays full prefill on the whole surrounding file

That is the design goal the llama.vim README states when it describes supporting very large contexts on low-end hardware through smart context reuse. Chat requests are minutes apart; completion requests are a second apart and nearly identical.

Question 4. Which of these actually demonstrate that your editor completions are not being sent to a third party? Select all that apply.

• A request appears in your local server log for every suggestion you see
• The server is bound to 127.0.0.1, confirmed with ss or lsof
• The extension description says "local"
• The extension's own telemetry setting has been read and set deliberately

Show the answer and why

Answer: A request appears in your local server log for every suggestion you see, The server is bound to 127.0.0.1, confirmed with ss or lsof, The extension's own telemetry setting has been read and set deliberately

The first two show where the model traffic goes and the last covers the separate question of what the extension itself sends. A description is a claim, not evidence, which is the distinction this whole course is built on.

Check my answersClear