How Does a Language Model Pick the Next Word?

You have almost certainly used a tool like ChatGPT, Claude, or Gemini. You type a sentence, and it types back something that reads like a person wrote it. The strange part is what happens inside: at every single step, the model is doing one thing — guessing the next word. That’s it. Everything else you’ve seen, from essays to code to bad poetry, comes out of that one move repeated thousands of times.

Words become numbers

Computers can’t read. So before anything else, your text gets chopped into tokens — chunks of words or word-pieces. “Explain quantum computing” might become `Explain`, `quant`, `um`, `comput`, `ing`. Each token maps to a list of numbers called a vector, and those numbers encode meaning picked up during training. Tokens that show up in similar contexts — “dog” and “puppy,” “run” and “jog” — end up with similar numbers.

The guess

The model runs your tokens through a very large neural network (billions of adjustable knobs called parameters, tuned during training). Out the other end comes a score for every token in its vocabulary — tens of thousands of scores, one per possible next token. Those scores get converted into probabilities.

If your prompt is “The capital of France is”, the model’s distribution might be 99% “Paris”, a sliver on “the”, and dust on everything else. If your prompt is “My favorite color is”, the top candidate might only get 8%, because there genuinely isn’t one right answer.

Temperature: the creativity dial

Here’s the part most people don’t know. The model doesn’t always pick the most likely word. It samples from that probability distribution, and a setting called temperature controls how adventurous the sampling is.

• Low temperature: nearly always take the top word. Safe, repetitive, precise.
• High temperature: spread out the odds. Surprising, varied — and more likely to be wrong.

This is why the same question can get different answers twice in a row. Nothing mystical is going on; it’s dice rolled over weighted probabilities.

Then it does it again

The chosen token gets appended to your text, and the whole process runs again for the next word. And again. A five-hundred-word answer is five hundred consecutive coin flips, each one informed by everything before it. When a model “hallucinates” a fake citation, it’s not lying — an earlier flip produced something plausible-sounding, and every later flip stayed consistent with it.

Why this matters

Understanding this changes how you use these tools. Ask for precision? Lower temperature-style prompting helps: give examples, demand structure, constrain the format. Ask for brainstorming? Loosen up. And when a model sounds confident while being wrong, remember: confidence was never evidence of truth. It was just the shape of the probability distribution.