How do LLMs work? Next-token prediction explained
By Unlearnt
A large language model, or LLM, is a neural network trained on large amounts of data to model language. Many chat models generate text by repeatedly predicting a next token, adding it to the context, then predicting again. That sounds small. Repeating it can produce a whole explanation, piece of code or conversation.
What does next-token prediction mean?
A token is a unit the model processes. It might be a word, part of a word, punctuation or another text fragment. The exact split depends on the tokenizer. Predicting the next token means assigning possible continuations different probabilities given the context so far.
Those probabilities are about continuations, not a verdict on whether a claim is true. A model can learn useful relationships while learning to predict language, but its generation process does not automatically check an external source.
A worked example: building one sentence
Suppose the prompt is: ‘Complete this sentence about making tea: Pour the hot water into the’. For this simplified example, treat each candidate word as one token. Imagine the model assigns ‘cup’ 55%, ‘teapot’ 30% and all other possibilities together 15%. These numbers are invented to show the mechanism, not measurements from a model.
If ‘cup’ is selected, it becomes part of the input for the next step. The model now continues ‘Pour the hot water into the cup’. It might next produce a full stop, or continue with ‘and’. It did not have to retrieve that entire sentence from a stored answer.
| Step | Context now includes | What happens next |
|---|---|---|
| 1 | …into the | Score possible next tokens. |
| 2 | …into the cup | Recalculate using the added token. |
| 3 | …into the cup. | Continue or end the response, depending on context and stopping rules. |
Does it always choose the most likely word?
No. Greedy decoding chooses the highest-scoring next token, while sampling can select from several possibilities. Temperature and other decoding settings alter how that selection behaves. A model can therefore produce different answers to the same prompt.
In the tea example, choosing ‘teapot’ is not necessarily an error. Several continuations can be sensible. More variation is not the same as more accuracy, and making generation less variable does not turn it into a fact-checker.
Training and answering are different jobs
During next-token pretraining, the model predicts continuations in training examples and adjusts its parameters to reduce prediction error. Further training can shape instruction following and other behaviours. When you use the resulting model, it generally computes an answer with those trained parameters rather than retraining them for each message.
Your prompt and conversation supply context for that answer. In transformer models, attention helps combine information from relevant positions in the available context. Adding ‘Use a teapot, not individual cups’ can therefore change the continuation without changing the model's underlying weights.
A common misconception: it searches the internet for every answer
A language model can generate an answer without any live web search. A product can connect it to search, documents or other tools, but that is an additional system capability. The returned material has to reach the model before it can inform the answer.
Likewise, remembering a detail within the current conversation is different from storing it for next week. An app can save information and supply it again later. That does not mean the base model learned the detail permanently during your chat.
What this helps you notice
If you ask about a fictional café's opening hours without supplying them, the model may still produce a neat answer. The tea example explains how a plausible continuation can be generated; it does not supply evidence about that café. Ask what information the system actually had, and whether a tool retrieved anything relevant.
When an answer goes wrong, separate missing context, unsupported claims and poor instructions. Those are different problems. Rewording a question might clarify your intent, but cannot make an unknown opening time become known.
Check your understanding
Optional: apply the idea to a new scenario.
You tell a chatbot that the fictional town of Bellwick has a purple library. Later in the same chat it correctly repeats the colour. Does this show that its training weights changed or that it searched the web? Explain how it could answer using next-token prediction, and what would be needed for a new chat to know the colour.
Answer with the tutor →Free to start, no account needed.