How do language models generate answers?

Text enters as tokens

Before generation begins, a tokenizer splits the instructions, conversation, tool results and other supplied context into tokens. A token may be a whole word, part of a word, punctuation or another text fragment. The model processes token identifiers, not sentences as people see them, so spelling, formatting and language can change how much context a prompt consumes.

The model predicts one next token

For the tokens seen so far, the network assigns a probability to each possible next token in its vocabulary. A decoding rule selects one token, adds it to the sequence and runs the prediction again. This loop continues until a stop token, a configured limit or another stopping condition is reached; the model does not normally fetch a complete answer that was stored in advance.

Context shapes the probabilities

System instructions, the user request, earlier messages and supplied documents all shift the next-token probabilities, but only while they fit in the active context. Decoding also matters: choosing the highest-probability token tends to be more repeatable, while sampling among plausible tokens can produce varied wording. Temperature and related controls change that selection process; they do not add facts or understanding.

Generation is not database retrieval

Training changes many distributed numerical weights so that patterns in text influence later predictions. Those weights are not a catalogue of source passages with reliable addresses. Unless a system separately retrieves documents or calls a tool, the model cannot look up a source record and prove where a statement came from. A fluent answer can therefore combine familiar patterns into a claim that has no factual support.

Practical limits to plan around

Models can invent details, follow an ambiguous instruction in an unintended way, miss information outside the context, repeat bias in their training material and make errors in calculation or multi-step reasoning. Treat important output as a draft: provide relevant context, request structured evidence, use retrieval or tools when current facts matter, and verify consequential claims against an independent source or test.

Questions people ask

Does a language model understand its answer?

It can represent complex relationships and produce useful reasoning-like text, but describing that as human understanding adds assumptions the mechanism does not establish. Operationally, it is predicting tokens from learned parameters and the current context.

Why can the same prompt produce different answers?

When decoding samples from several plausible next tokens, an early different choice changes every probability that follows. Deterministic settings reduce variation, but they do not guarantee that the repeated answer is correct.

Can a model quote its sources?

Only when sources are supplied through context, retrieval or a tool and the system preserves that connection. A citation generated from model weights alone may look convincing and still be invented, so verify it before relying on it.