How LLMs Actually Work

Why you need a mental model

You don't need to understand neurons and linear algebra. But having a basic idea of how LLMs work will help you write significantly better prompts. It's like driving — you don't need to be a mechanic, but it helps to know that fuel goes in the tank and not in the windshield washer reservoir.

Most prompting mistakes stem from misconceptions about what AI is and how it works. People expect AI to 'understand' their intent, 'remember' previous conversations, or 'know' current information. Once you understand reality, you stop making these mistakes and your prompts dramatically improve.

LLM = the world's most advanced autocomplete

Large Language Models work on one fundamental principle: they predict the next word. More precisely, the next token. All the 'magic' is sophisticated prediction: based on what you wrote, the model estimates what should come next. It doesn't plan ahead, it has no hidden agenda — it generates text token by token.

This doesn't mean the outputs are simple. Modern models have hundreds of billions of parameters and were trained on trillions of tokens of text. The result is a system that produces surprisingly coherent, useful, and often brilliant responses — but still based on prediction, not understanding.

Tokens: how AI reads your text

A token isn't a word. It's a chunk of text — it might be a whole word, part of a word, or even just a character. English 'hello' is one token. The word 'unbelievable' might be two or three tokens. Numbers get tokenized digit by digit. Why does this matter? Because AI limits are measured in tokens, not words — and some languages are 'more expensive' than others in terms of tokens per word.

Modern models have a context window — the total number of tokens they can work with at once. This includes your prompt AND the AI's response. GPT-4o has 128K tokens, Claude Sonnet has 200K tokens, with the latest models handling a million or more. When you exceed the limit, the model 'forgets' the beginning of the conversation — information at the start gets lost.

Temperature and top-p: why AI answers differently each time

Ever noticed that the same prompt gives a slightly different answer each time? That's not a bug — it's by design. The 'temperature' parameter controls randomness. Low temperature (0-0.3) = more consistent, predictable responses — ideal for factual tasks, code, analysis. High temperature (0.7-1.0) = more creative, diverse responses — ideal for brainstorming, creative writing, idea generation.

The attention mechanism: not all tokens are created equal

LLMs use a mechanism called 'attention.' When generating each token, the model 'weighs' all previous tokens — but not equally. It pays more attention to some than others. Practically, this means: instructions at the beginning and end of your prompt typically have more influence than those in the middle. This is known as the 'lost in the middle' effect.

What this means for prompting

Knowing how the model works directly affects your prompting strategy. When you know the model predicts the next token, you understand why clear structure helps — it steers prediction in the right direction. When you know the model has no memory between conversations, you stop expecting it to 'remember' last week's chat. When you know about the 'lost in the middle' effect, you put key instructions at the beginning and end of your prompt.

Three rules that follow from this

First: be explicit, not implicit. The model won't guess what's in your head — it works only with what you write. Second: order matters. The model processes text left to right, token by token. Put important instructions first. Third: when the response isn't good, the problem is almost always in the prompt, not the model. Reformulate instead of getting frustrated.