How LLMs Actually Work

Why you need a mental model

You don't need to understand neurons and linear algebra. But having a basic idea of how LLMs work will help you write significantly better prompts. It's like driving — you don't need to be a mechanic, but it helps to know that fuel goes in the tank and not in the windshield washer reservoir.

Most prompting mistakes stem from misconceptions about what AI is and how it works. People expect AI to 'understand' their intent, 'remember' previous conversations, or 'know' current information. Once you understand reality, you stop making these mistakes and your prompts dramatically improve.

LLM = the world's most advanced autocomplete

A language model generates text by predicting further tokens from context. Modern products may also use internal reasoning, planning and tools. Autocomplete is a useful simplified analogy, not a complete description of every current system.

This doesn't mean the outputs are simple. Modern models have hundreds of billions of parameters and were trained on trillions of tokens of text. The result is a system that produces surprisingly coherent, useful, and often brilliant responses — but still based on prediction, not understanding.

Tokens: how AI reads your text

A token can be a word, part of a word, a character or a group of digits. Splitting depends on the tokenizer; numbers are not always split digit by digit. Check counts with the relevant provider’s tokenizer. Word-to-token ratios are rough estimates, especially across languages.

Context capacity depends on the specific model, product and plan; check current documentation and allow room for output. A chat may truncate or summarize older messages, or reject an oversized input. A large window does not guarantee accurate use of every detail.

Temperature and top-p: why AI answers differently each time

For models that support it, temperature changes randomness in token selection. A lower value does not guarantee determinism or factual accuracy. Check the supported range and parameters for the specific model; some reasoning models restrict them.

The attention mechanism: not all tokens are created equal

LLMs use a mechanism called 'attention.' When generating each token, the model 'weighs' all previous tokens — but not equally. It pays more attention to some than others. Practically, this means: instructions at the beginning and end of your prompt typically have more influence than those in the middle. This is known as the 'lost in the middle' effect.

What this means for prompting

Knowing how the model works directly affects your prompting strategy. When you know the model predicts the next token, you understand why clear structure helps — it steers prediction in the right direction. When you know the model has no memory between conversations, you stop expecting it to 'remember' last week's chat. When you know about the 'lost in the middle' effect, you put key instructions at the beginning and end of your prompt.

Three rules that follow from this

First: be explicit, not implicit. The model won't guess what's in your head — it works only with what you write. Second: order matters. The model processes text left to right, token by token. Put important instructions first. Third: when the response isn't good, the problem is almost always in the prompt, not the model. Reformulate instead of getting frustrated.