Large‑language models (LLMs) generate a response one token at a time by repeatedly asking the same question: “What token is most likely to come next given everything that has already been produced?”
- Tokenisation – The input prompt is first broken into discrete sub‑word units (tokens) and each token is turned into a numerical embedding that captures its meaning in a high‑dimensional space [1][2].
- Contextual encoding – All previously generated tokens (including the original prompt) are fed through a transformer network. The self‑attention layers let the model weigh the importance of each token relative to the others, producing a contextual representation of the whole sequence [1][3].
- Probability distribution – The final hidden state is passed through a linear layer and a soft‑max function, yielding a probability distribution over the entire vocabulary for the next token [4][5].
- Sampling / decoding – The model selects the next token from that distribution. Common strategies are:
- Greedy decoding – pick the highest‑probability token.
- Temperature sampling – soften or sharpen the distribution before random sampling.
- Top‑k / top‑p (nucleus) sampling – restrict sampling to the most likely k tokens or to the smallest set whose cumulative probability exceeds p.
- Beam search – keep multiple candidate sequences and expand them in parallel, later choosing the one with the best overall score.
- Iterative rollout – The chosen token is appended to the output sequence, the transformer recomputes the contextual representation with the new token added, and steps 2‑4 repeat until a stopping condition is met (e.g., an end‑of‑sentence token, a max‑length limit, or a user‑defined stop signal) [4][5].
- Fixed computation per token – Each token generation passes through the same number of neural‑network layers, so the amount of processing required does not grow with the difficulty of the question; the model’s “thinking time” is constant for every token [6][7].
In practice, the next‑token predictor learned from massive text corpora (hundreds of billions of tokens) captures statistical regularities of language rather than explicit rules [8][4]. By chaining together many such predictions, the LLM produces coherent, context‑aware answers.