Prefill vs decode: how an LLM builds a reply

An LLM builds a reply in two stages. Prefill processes the prompt and produces the first reply token. Decode continues the reply, generating one new token at each step.

A token is a small piece of text. It can be a word, part of a word, or punctuation. We'll use familiar words to show the steps, although a real tokenizer may split them differently.

This is Part 2 of the AI Engineering Interview Series.

Watch: Prefill vs decode on YouTube · Discuss: The Day 2 LinkedIn post

Start with one email

Suppose your request is:

Reply politely to this email: “Can we meet tomorrow?”

The instruction and the email together form the prompt. Both are available before generation begins. The model needs the instruction to know that it should write a polite reply, and the email to know what that reply should address.

Suppose the final answer will be:

Yes, tomorrow works for me.

That sentence is built through prefill followed by repeated decode steps. Follow what is available at each stage: first the prompt, then the prompt plus a growing reply.

Prefill processes the prompt and starts the reply

During prefill, the model processes the instruction and the email. Because all the prompt tokens are already available, it can process multiple positions together. It still respects the text's order.

At the end of prompt processing, the model produces scores for possible next tokens. One token is selected to begin the reply. Suppose it is “Yes”.

At this point, the model has processed the request and started answering. It hasn't produced the complete sentence. The next stage continues from that first token. NVIDIA's explanation of LLM inference describes this split between prompt processing and subsequent token generation.

Decode continues the reply one token at a time

The first decode step takes “Yes” as the new input. Using the earlier prompt as context, the model works out which token should follow it.

Suppose the next token is a comma. The reply now reads “Yes,”. That comma becomes the new input for the following step, which chooses the next token. Each selected token is fed back into the model so it can continue the answer.

At each step, the model uses both the original request and the reply generated so far. This is why the continuation can stay relevant to the email and fit the words that precede it. It isn't choosing each word in isolation.

The reply gradually grows:

Yes
Yes,
Yes, tomorrow
Yes, tomorrow works
Yes, tomorrow works for me.

These lines show progress rather than exact tokenizer boundaries. Several decode steps may be needed to produce one displayed word. The model continues until it reaches a stopping condition, such as an end-of-answer token or a configured output limit.

The dependency is what matters: the next step uses the token selected by the previous step. The reply therefore develops in sequence. PyTorch's explanation of decoding describes this repeated generation process.

Compare what each stage does