Act one: How a model is made
This happens once. It takes months, an enormous amount of text and a data center full of processors, and it produces a single artifact: a file of numbers. Nothing in this act happens while you are using the model. Terms shown in gold open a definition, here and everywhere else on the page.
Stage 01 · The training data
Text at a scale no person could read
Before anything can be trained there has to be text, and the quantity involved is the first thing that breaks most people's intuition. A large model is trained on a body of text that would take a person reading continuously for many thousands of years to get through.
In one sentence
Gather an extremely large body of text, clean it, remove duplicates, and turn it into the raw material the model will learn from.
Why it matters
Everything the model knows in its own weights arrives here. No later training stage meaningfully adds facts to them. If something was well represented in this text the model will tend to handle it well, and if it was rare, wrong or absent, the model will tend to be weak on it in ways that are hard to predict from the outside.
That is a claim about the weights, not about the product. A finished assistant can be handed fresh information at the moment you ask, through search, retrieval or a tool call, and it will use it. What that information never does is change the model. It arrives in the context window and it is gone at the end of the conversation.
This is also the least visible part of the whole process. Model developers publish detailed papers about architecture and comparatively little about exactly what they trained on.
· Where the text comes from
Web crawl, books, code, reference works
Four broad sources, mixed in proportions that vary by developer and are rarely published in full.
The sources
The public web. common crawl is the largest openly available example, and it is filtered heavily before use because most of the web is not text you would want a model to imitate.
Books and long form writing. Valuable out of proportion to their size, because a book sustains an argument across hundreds of pages and teaches the model what long, coherent reasoning looks like.
Source code. Public repositories. Code turns out to improve performance on tasks that are not code, which is one of the more interesting empirical findings in the field and is not fully explained.
Reference and academic material. Encyclopedias, papers, technical documentation. Dense, edited and unusually reliable per token.
The honest caveat
Exact training set composition is generally treated as commercially sensitive and is not disclosed. Any specific claim about what a given commercial model was trained on should be treated as an estimate unless the developer published it.
· Filtering and deduplication
Most of what is collected gets thrown away
The gap between text that exists and text worth training on is enormous, and closing it is a large part of the actual engineering work.
What gets removed
Boilerplate, navigation menus, spam, machine generated filler, adult and violent content, and text in languages outside the target set. What survives is usually a small fraction of what was collected.
Deduplication matters more than it sounds. The same passage appearing thousands of times across the web gets weighted as though it were thousands of independent pieces of evidence, and near duplicate text is one of the routes by which a model ends up able to reproduce a passage close to verbatim.
What filtering does not remove
Filtering targets material that is unusable or harmful. It does not, and cannot, remove the ordinary distribution of the text itself. Whose writing is over represented, which languages are thin, which occupations get described with which adjectives, which viewpoints are common and which are rare, all of that survives filtering and is learned.
So the model's default assumptions are the training set's default assumptions. Later stages can suppress specific outputs, and they do not reach back and change what was absorbed. This is the origin of most of what gets discussed as model bias, and it starts here rather than at the rlhf stage where it usually gets blamed.
Bias in training data deserves fuller treatment than one paragraph. It is a first order topic, and this page treats it as an introduction rather than the full account.
· The unsettled question of rights
Live litigation, no settled answer
Whether training a commercial model on copyrighted material without a license is permitted is being contested in courts in several jurisdictions, and it is not resolved.
Where this stands
There are active lawsuits from authors, publishers, news organizations and rights holders. There are also licensing agreements between AI developers and publishers, which tells you the developers do not regard the question as free of risk.
This page does not take a position on the legal question. It records that the question is open, because an explainer that presents training data as a neutral technical input is leaving out the part that is currently in dispute.
Stage 02 · Pre-training
One task, repeated a staggering number of times
pre training is the phase that produces almost everything the model can do, and it is trained on a single task that sounds far too simple to work: given some text, predict what comes next.
In one sentence
Show the model an enormous amount of text with the next piece hidden, let it guess, measure how wrong the guess was, and adjust its numbers slightly to be less wrong. Repeat until the budget runs out.
Why such a plain task produces so much
To predict the next word in a legal contract you have to have absorbed how contracts are written. To predict the next line of a proof you have to have absorbed something about how proofs go. To finish a sentence that begins with the capital of Australia is, you have to have stored that Canberra follows.
None of that was taught deliberately. It is what prediction forces, given enough text and enough capacity. It remains one of the most surprising empirical results in the field, and the reason so much investment followed.
What comes out the other end
A model at the end of pre-training is not a chatbot. Given the words `Explain the causes of the First World War`, it is as likely to continue with more exam questions as to answer, because in its training data that phrasing often appeared on a list. Turning it into something that answers is the job of the next stage.
· The prediction task
Next token prediction, worked through
The model never sees words. It sees tokens, and it produces a score for every token in its vocabulary describing how good a continuation each one would be.
A worked example
Take the fragment `The capital of France is`. The model computes a score for every one of the tens of thousands of tokens it knows.
input : The capital of France is
scores : ' Paris' {{hl:18.4}}
' the' 16.1
' located' 15.3
' a' 13.9
' Lyon' 9.2
... (tens of thousands more)
{{cm:correct answer in the training text was ' Paris'}}
{{cm:error is measured, weights are nudged, next example loads}}
{{cm:scores are illustrative. real values depend on the model.}}
The scale of the repetition
That loop runs across trillions of tokens. No individual pass teaches the model much. The capability comes from the accumulation.
· Why it is called self-supervised
No labels, no annotators
Training normally needs labeled examples, which means people producing them. Next token prediction avoids that entirely.
The trick
The correct answer is already sitting in the text. Hide the next token and you have a question with a known answer, and you can manufacture as many of those as you have text. That is why this approach scaled when earlier approaches could not. The bottleneck stopped being human annotation and became compute and data.
· Parameters and what a model file is
The whole thing is a table of numbers
A trained model is a file. Not a program with logic in it, not a database of facts, and not a compressed copy of the internet. A file of numbers.
What is in the file
Every connection in the neural network has a weight, a number controlling how much influence one unit has on the next. A model described as 70 billion parameters has 70 billion such numbers. Pre-training is the process of finding good values for all of them.
Because the file is only numbers, it can be copied, shipped, quantized to make it smaller, and run offline. It is also why a model cannot be edited the way a document can. There is no line to find and change.
The proportion people get wrong
Parameter count is closer to engine displacement than to intelligence. Data quality, training duration and post-training all move capability substantially at a fixed parameter count.
This is not folklore. The 2022 paper that introduced the model known as Chinchilla showed that the large models of the time were substantially undertrained for their size, and that a considerably smaller model trained on far more data outperformed them. The finding reset how the industry allocates a training budget between size and data.
· How learning actually happens
Backpropagation and gradient descent
backpropagation is the mechanism that turns a wrong answer into a slightly better set of numbers. It is the single idea without which none of this exists.
The four steps, repeated
Forward. Text goes in, the model produces its scores.
Loss. The scores are compared with what actually came next, giving one number describing how wrong the model was.
Backward. Working from the output back toward the input, the method calculates how much each individual weight contributed to that error. This is the step that gives the technique its name.
Update. Every weight moves a small distance in the direction that would have reduced the error. The size of that step is the learning rate.
The mental picture
Picture a landscape with a low point somewhere in it, where the height at any position is how wrong the model is. Backpropagation tells you which way is downhill from where you are standing. Gradient descent is taking a step that way. Do it enough times and you end up somewhere low.
The landscape here has as many dimensions as the model has parameters, so the picture breaks down quickly. The procedure does not.
· Why this was hard to arrive at
The credit assignment problem
The obstacle that held the field back for years was not the idea of a layered network. It was working out which weight to blame.
The problem
In a network many layers deep, a wrong answer at the output is the accumulated result of millions of contributions. Assigning responsibility to each one, efficiently enough to do it billions of times, is the whole difficulty. That is what reverse mode differentiation solves, and it is the reason the history section spends time on who worked it out and when.
Stage 03 · Post-training
Turning a text predictor into an assistant
post training is everything done after pre-training to make the model follow instructions and behave acceptably. It is a small fraction of the compute and a very large fraction of what you experience.
In one sentence
Take the raw predictor and shape its behavior, using examples of good answers and then human judgments about which answers are better.
The distinction worth holding on to
Pre-training gave the model its capability. Post-training gives it its manners, its format, its refusals and its tone. When two assistants built on similar underlying models feel different to use, most of that difference was made here.
It also means the personality is a designed artifact. It was chosen by a company and implemented by contractors, and it could have been chosen differently.
· Supervised fine-tuning
Showing it what a good answer looks like
fine tuning continues training on a much smaller, deliberately written set of examples, each one an instruction paired with a good response.
How it works
The mechanism is identical to pre-training. Same loss, same backpropagation, same weight updates. Only the data changes: instead of arbitrary web text it is curated demonstrations of the behavior wanted.
The volume is small by comparison, often tens of thousands of examples rather than trillions of tokens, and it is enough because the underlying ability is already there. This stage is directing an existing capability, not creating one.
The common misconception
Fine-tuning is regularly proposed as the way to make a model know a company's internal documents. It generally works poorly for that. It reliably changes style, format and task behavior, and it does not reliably install retrievable facts. Putting the documents in front of the model at question time works better.
· Reinforcement learning from human feedback
Ranking answers, and what that encodes
rlhf addresses a problem fine-tuning cannot: for most questions there is no single correct answer to demonstrate, but people can still say which of two answers is better.
The three steps
Collect preferences. The model produces several answers to the same prompt. People rank them.
Train a judge. Those rankings train a separate model whose only job is to predict which answer a person would prefer.
Optimize against the judge. The main model is then adjusted to produce answers that score well with the judge.
What this actually encodes
Better is not a technical property. It is defined by the guidelines the company writes and by the judgment of the people it hires to apply them. Helpfulness, harmlessness, refusal boundaries and tone all get set here.
This is the most underexamined step in public discussion of these systems. A great deal of what people describe as the model's values was decided in a rating interface.
· The reward model
A second model that scores the first
The judge trained in step two is itself a neural network, and its limits become the main model's limits.
Why this creates a known failure mode
Optimizing hard against an imperfect judge produces answers that score well without being good. Length, confidence and a well organized surface are easy to reward and easy to overproduce.
This is the standard explanation for why assistants tend toward long, agreeable, heavily formatted answers. Those traits scored well. The technical term for pushing a measure until it stops measuring what it stood for is reward hacking, and it is an active area of work rather than a solved problem.
· Newer and related methods
DPO, AI feedback, and where this is going
The three step recipe above is the classic form. Several variants are now in common use.
The main ones
Direct preference optimization skips the separate judge and adjusts the model straight from the preference pairs. Simpler and cheaper, widely adopted.
AI feedback. Rather than a person ranking every pair, a model does the ranking against a written set of principles, with people setting and auditing the principles. This is how preference data gets produced at a volume human raters could not reach.
Reinforcement learning against checkable answers. Where correctness can be verified automatically, as in mathematics or code that either runs or does not, the model can be trained against that signal directly. This is the main engine behind recent gains in reasoning.
· What actually ships
The weights are not the product
What a user talks to is never just the model file. Several distinct things sit between the two.
The layers around it
The weights. The output of everything above. Static from here on.
The system prompt. Standing instructions placed ahead of the conversation, setting the persona, the rules and the format.
The harness. The software that assembles the context, runs the model, executes any tool requests, applies safety filtering, and streams the result back.
The harness is where most product behavior lives. Memory features, document retrieval, web access and agent behavior are all harness, not model. Confusing the two is the source of a large share of misplaced expectations in both directions.
Act two: How a model answers
This is inference, and it happens every time you press send. It takes seconds and it changes nothing about the model. Follow it from your typed sentence to the words appearing on screen. Each stage opens, and most go two or three levels deeper.
Step 01 · Tokenization
Your sentence becomes numbers
The model cannot read letters. The first thing that happens to anything you type is that it gets cut into tokens and each token is replaced by its index in a fixed list.
In one sentence
Chop the text into known pieces and look up the number for each piece.
What it looks like
you type : Tokenization is unfamiliar.
tokens : {{hl:["Token"]["ization"][" is"][" unfam"]["iliar"]["."]}}
ids : [30642, 1634, 318, 45964, 15516, 13]
{{cm:6 tokens for 4 words. common words are single tokens,}}
{{cm:rarer ones get assembled from pieces. spaces belong to tokens.}}
{{cm:ids shown are illustrative. every model has its own vocabulary.}}
Two consequences you can feel
Cost and length limits are counted in tokens, so an English sentence, the same sentence in Thai, and a block of code with the same character count can differ by several times in what they consume.
And a model that never sees individual letters is working uphill on anything letter shaped. Counting the letter r in a word, or reversing a string, are hard for a system whose smallest unit is a chunk of several characters.
· The vocabulary
A fixed list, frozen before training
The vocabulary is decided before training begins and cannot change afterward.
Why frozen matters
Every one of the model's output scores corresponds to one entry in this list. Adding an entry later would mean the model has no learned weights for it, so the list is fixed at the start and lives with the model for its whole life.
Unfamiliar input is generally still representable, because it gets built from smaller pieces already in the list. Most current tokenizers work over bytes rather than characters, so there is a floor they can always fall back to. It costs more tokens, which is why unusual names, rare languages and dense notation consume disproportionate context.
· Byte pair encoding
How the list gets built
bpe is the method that decides which pieces make the list, and nobody writes the rules by hand.
The algorithm, in full
Start with every text as individual characters. Count every adjacent pair. Merge the most frequent pair into a single new unit. Count again. Repeat until you have as many units as you want.
That is the entire method. It was published as a data compression technique in 1994 and borrowed for language modeling twenty one years later.
Why it wins
One procedure handles English, Japanese, Python and emoji with no language specific rules, because it only ever counts what is frequent in the text it is given. Frequent sequences become single units and rare ones stay in parts, which is exactly the tradeoff you want.
· Watch it merge
Four rounds on a tiny corpus
On a corpus of the words low, lower, newest and widest, repeated at different frequencies.
Round by round
start : l o w l o w e r n e w e s t w i d e s t
round 1 : most frequent pair is {{hl:e s}} -> merge to 'es'
l o w l o w e r n e w es t w i d es t
round 2 : most frequent pair is {{hl:es t}} -> merge to 'est'
l o w l o w e r n e w est w i d est
round 3 : most frequent pair is {{hl:l o}} -> merge to 'lo'
lo w lo w e r n e w est w i d est
round 4 : most frequent pair is {{hl:lo w}} -> merge to 'low'
low low e r n e w est w i d est
{{cm:'est' emerged as a unit without anyone stating that it is a suffix.}}
{{cm:run this for tens of thousands of rounds on real text and you}}
{{cm:get a working vocabulary.}}
· Why token counts drive cost
The unit of billing and of memory
Tokens are the unit in which almost everything about running these models is measured.
What it affects
Price is per token, in and out. The context window is a token limit. Latency scales with tokens produced, because they are produced one at a time.
A practical consequence: a long document pasted into a conversation is not free and does not go away. It stays in the context and is resubmitted with every subsequent message in that conversation.
Step 02 · Embeddings
Numbers that carry meaning
A token id is just a lookup number, and 30642 is no closer to 30643 in meaning than to 8. embeddings fix that by replacing each id with a long list of numbers whose position in space reflects how the token gets used.
In one sentence
Swap each token's arbitrary id for a learned list of numbers, positioned so that tokens used in similar ways sit near each other.
Where the meaning is
The idea underneath is old and comes from linguistics: a word is characterized by the words that occur around it. If two tokens keep turning up in the same kinds of company, they end up in the same region of the space, and this happens without anyone writing down what any word means.
Nothing here is a dictionary. There is no definition stored anywhere. There is only position, and position is derived entirely from patterns of co-occurrence in the training text.
· What a vector actually is
A direction in a space with too many directions
A vector is a list of numbers, and a list of numbers is a position.
Building up
Two numbers place a point on a map. Three place it in a room. Three hundred place it in a space you cannot picture, and the arithmetic works identically no matter how many you use.
The main models reported in the 2013 Word2Vec papers used 300 numbers per word, and that is also the size of the vectors Google released. The largest model in the GPT-3 paper uses 12,288 numbers per token, stated in that paper's own table of model sizes.
Measuring closeness
Similarity is measured by the angle between two vectors rather than the distance between them, a measure called cosine similarity. Two tokens pointing in nearly the same direction are treated as similar regardless of how far out along that direction they sit.
· The famous analogy, honestly
King minus man plus woman
The best known demonstration of embedding geometry is also the one most often overstated, so it is worth separating what is solid from what is fragile.
What is solid
Embedding spaces really do encode relationships as directions. There is a consistent direction that relates singular to plural, another that relates a country to its capital, another that relates a verb to its past tense. This is a real and repeatedly confirmed property, and it is the reason embeddings are useful.
What is fragile
The specific result where king minus man plus woman lands on queen depends on the scoring code deleting the three input words from the list of candidate answers before it picks the closest match. Put them back and the top answer is usually one of the words you started with, which for this example is king.
The evidence and the sources sit in the history section under Word2Vec. The geometry survives the correction. The party trick does not survive it intact.
· Positional encoding
Otherwise word order does not exist
positional encoding is the part almost every popular explanation skips, and leaving it out makes the next stage impossible to understand properly.
The problem
The attention mechanism has no built in sense of sequence. Handed a set of embeddings it treats them as an unordered collection. The dog bit the man and the man bit the dog would arrive as the same input.
The fix
Position information is mixed into each token's embedding before the token enters the stack, so a token carries both what it is and where it sits.
How that is done has turned out to matter a great deal in practice. It is one of the main factors in whether a model behaves sensibly on inputs longer than anything it saw during training, which is why context length claims deserve some skepticism until tested.
Step 03 · The transformer stack
Where context gets built
This is the engine. The same block of machinery, repeated dozens or hundreds of times, each repetition taking the previous one's output and enriching it further. It is the part that earns the word transformer.
In one sentence
Every token looks at the tokens before it, pulls in what is relevant, then gets processed on its own. Then it happens again, and again.
The block
One block is two components. attention moves information between tokens. The feed forward network then transforms each token on its own. Around both sit residual connections and normalization, which are what make a stack this deep trainable at all.
Stack the block many times and each pass operates on a representation that is already richer than the last. This is where a token stops standing for a piece of text and starts standing for that piece of text in this particular context.
The claim to be careful with
You will often read that early layers handle grammar, middle layers handle meaning and late layers handle reasoning. There is real evidence for some depth related specialization, and the tidy three way split is a simplification that interpretability researchers do not treat as established. It is a useful picture. It is not a finding.
· Self-attention
The mechanism that does the work
The core operation. For each token, look at every token available, decide how relevant each one is, and pull in a weighted blend of what they carry.
The example that makes it click
Take the word bank in two sentences. In the bank of the river was muddy, the tokens river and muddy are highly relevant and the representation of bank shifts toward geography. In I went to the bank to deposit money, deposit and money are the relevant ones and it shifts toward finance.
Same starting embedding. Different neighbors. Different resulting representation. That is the entire trick, and it is why these models handle ambiguity that defeated every rule based system before them.
What it is not
Attention is a weighted average computed with matrix arithmetic. The name invites a comparison with human attention that does not survive contact with the mechanism. Nothing is being focused on and nothing is being noticed.
· Query, key and value
Three roles every token plays
Each token produces three different vectors from its own representation, and the names describe what each is for.
The three
Query. What this token is looking for.
Key. What this token offers to anything looking.
Value. What actually gets passed along when a match occurs.
Compare one token's query against every token's key. Where a query and a key line up you get a high score. Turn those scores into weights with softmax, then take the weighted blend of the values. That blend is what the token carries forward.
The library picture
Your query is what you want. The keys are the spine labels. The values are the contents. You compare your query against every spine, and you leave with a blend of the contents weighted by how well each spine matched, rather than with one book.
· Multi-head attention
Many attentions at once
Multi-head attention runs the whole operation many times in parallel with separate weights, and combines the results.
Why more than one
Relevance is not one thing. Which noun a pronoun refers to, which verb a subject belongs to, which earlier clause a phrase modifies, and plain topical relatedness are different relationships and a single weighted average cannot represent all of them at once.
Running many heads lets different ones become sensitive to different relationships. No one assigns those jobs. Whatever specialization appears is a product of training, and working out what any given head has ended up doing is an active research problem rather than a lookup.
· Causal masking
Why a generator can only look backward
In a model built to generate text, each token is allowed to attend to tokens before it and forbidden from attending to tokens after it.
Why the restriction exists
Training works by hiding the next token and asking the model to predict it. If attention could see ahead, the answer would be visible in the input and the model would learn nothing.
The restriction is enforced by setting the scores for later positions to negative infinity before softmax, which drives their weight to zero. It is not a rule imposed afterward. It is built into the arithmetic.
This is also why generation is inherently left to right and why the autoregressive loop looks the way it does. Models built for classification rather than generation drop the mask and can see the whole input at once.
· The feed forward network
The quiet half that holds most of the parameters
After attention has gathered context, each token's numbers pass alone through a small two layer network. No token sees any other at this point.
What it contributes
Attention decides what is relevant. This step does something with it. The layer expands each token's representation to several times its width, applies a nonlinearity, then compresses it back.
In the standard design, where the expansion is fourfold, this component holds roughly two thirds of the parameters in every block. Interpretability work indicates a substantial amount of factual recall lives here rather than in attention, which makes the quieter half arguably the more mysterious one.
· Residuals and normalization
Why a deep stack trains at all
Two pieces of engineering that get no attention in popular accounts and without which none of this works.
Both, briefly
Residual connections. Each component adds its output to its input rather than replacing it. This gives the error signal a clear path back through the whole stack during backpropagation, and it is the main reason networks hundreds of layers deep can be trained. Without it the signal degrades with depth until learning stalls.
Normalization. The numbers flowing through are rescaled to a consistent range at each step, which stops values exploding or collapsing across a long stack.
Neither adds capability in any interesting sense. Both are what make the depth survivable, and depth is where the capability comes from.
· What stacking buys
Progressive refinement
A single block is not powerful. The power comes from repetition, and each pass works on a richer input than the last.
The progression
After one pass, a token's representation reflects its immediate neighbors. After several, it reflects relationships between those neighbors. After many, it can reflect structure spanning the whole input.
Where a model has, say, eighty blocks, the representation at the top is eighty rounds of refinement away from a plain lookup. That distance is the difference between a table of word vectors and a system that can follow an argument.
Step 04 · Choosing the next word
Scores, probabilities and a roll of the dice
The stack finishes and produces one score for every token in the vocabulary. Turning that set of scores into a single chosen token is a separate process, and it is where every setting people tune actually lives.
In one sentence
Convert raw scores into probabilities, narrow the field, then pick one at random from what remains.
The part that surprises people
The model does not choose a word. It produces a probability for every word, and software outside the model draws a sample from that distribution. Ask the same question twice and get different answers, and this is why. Nothing changed in the model. A different sample was drawn.
· Logits
The raw output
logits are the unprocessed scores. One per vocabulary entry, higher meaning a better continuation, on no fixed scale.
What they are and are not
They are not probabilities. They can be negative and they do not sum to anything meaningful. Only their relative ordering and the gaps between them carry information.
This is the real output of the model. Everything from here on is a choice made by the software running it, and that software could make different choices without touching a single parameter.
· Softmax
Scores into probabilities
softmax converts the whole set of logits into probabilities that sum to one.
What it does to the gaps
It exponentiates each score and divides by the total, which preserves the ordering and stretches the differences. A leading token with a modest edge in score comes out with a large edge in probability.
That stretching is the reason models sound confident. Even where the top two candidates were nearly tied on raw score, the resulting distribution usually is not close.
logits softmax probabilities
Paris 18.4 {{hl:0.86}}
the 16.1 0.09
located 15.3 0.04
a 13.9 0.01
Lyon 9.2 0.00
{{cm:------}}
1.00 {{cm:always sums to one}}
{{cm:a gap of 2.3 in raw score becomes a gap of 0.86 against 0.09.}}
{{cm:figures are illustrative and computed from the logits shown.}}
· Temperature
How much of the tail stays in play
temperature is applied to the logits before softmax and changes how sharp or flat the resulting distribution is.
The range, described honestly
At or near zero. The distribution collapses onto the single highest scoring token. The same prompt gives the same answer every time. Repetition and loops are common.
Around 0.7 to 1.0. The usual working range. Leading candidates dominate, alternatives retain a real chance.
Above 1.2. The distribution flattens, low scoring tokens become live options, and output drifts from surprising to unreliable to incoherent.
What it does not do
Temperature does not make a model more creative, more truthful or more careful. It changes how much probability is left on options the model rated lower. A high temperature answer is not a bolder answer. It is an answer that took a less likely path.
· Top-p and top-k
Cutting the field before sampling
Two ways of discarding the long tail before a sample is drawn. Both are usually applied alongside temperature.
The two
Top-k. Keep the k highest scoring tokens and discard the rest. Fixed count, insensitive to how confident the model was.
top p. Rank the candidates, add their probabilities from the top down until the total reaches p, keep those and discard the rest. The size of the surviving set changes with the model's confidence, which is why this is generally preferred.
The misreading to avoid
Top-p at 0.2 does not mean the top 20 percent of tokens. It means the smallest set of tokens whose probabilities add up to 20 percent, which where the model is confident can be a single token. It is a share of probability, not a share of the list.
Step 05 · The autoregressive loop
One token, then everything again
One token has now been chosen. To produce the second, the entire process runs again with that token appended to the input. autoregressive generation is that loop, and almost every characteristic behavior of these systems follows from it.
In one sentence
Append the chosen token to the input and run the whole thing again. A paragraph is a few hundred repetitions.
The loop
pass 1 in: The capital of France is
out: {{hl:Paris}}
pass 2 in: The capital of France is Paris
out: {{hl:,}}
pass 3 in: The capital of France is Paris,
out: {{hl:and}}
{{cm:each pass conditions on everything before it. the model has no}}
{{cm:working notes, no draft, and no intention about where this is going.}}
One engineering correction, because the picture above misleads on cost
Conceptually every step depends on the whole sequence so far. Mechanically, production systems do not recompute the whole prefix each time. They cache the intermediate values already calculated for earlier tokens and compute only the new position, which is the reason generation is affordable at all.
So the honest statement is that the model carries no state between your requests, while within a single response the software running it carries a great deal of state as an optimization. Reading the loop as a full re-read from scratch every time will give you the wrong intuition about latency, about cost, and about why long prompts get cached.
Sit with this one
The model has no plan. When it starts a sentence it does not know how the sentence ends. Every coherent paragraph you have read from one of these systems was produced one blind step at a time, with each step conditioned only on what was already on the page.
· Why this explains hallucination
The failure mode falls out of the design
hallucination is not a defect bolted onto an otherwise reliable system. It is the same mechanism working exactly as built, in a place where nothing anchors it.
The mechanism
At every step the system selects a likely continuation. Where it has strong grounding, likely and true tend to coincide. Where it does not, likely is still available and the loop still has to produce something.
A plausible looking citation is a likely continuation. So is a confident date. The machinery cannot distinguish between recalling and constructing, because from the inside both are the same operation.
What follows for anyone using one
Fluency carries no information about accuracy. The confident tone was set during post training and applies uniformly, whether the model is on solid ground or not.
The measures that work are structural: give the model the actual source material, ask for citations you can check, and verify anything load bearing. Asking a model to be accurate does not address the mechanism, because at no point in the loop is accuracy the quantity being computed.
· Why it is slow, and why it streams
The one at a time constraint
Reading your prompt can be done in parallel. Writing the answer cannot.
The asymmetry
The whole point of the transformer was that a sequence can be processed all at once, which is what made training at scale possible. Generation cannot use that, because the second token depends on the first having been chosen.
So a long answer takes proportionally longer, and text appearing word by word on screen is not a presentation effect. It is the loop, running.
It also means output tokens cost more to produce than input tokens, which is why pricing usually separates the two.
Step 06 · The context window
Everything the model can see
The context window is the total amount of text the model can have in front of it at once. It is the single most misunderstood limit in these systems, because people reach for the word memory and the word does not fit.
In one sentence
A fixed size working surface holding the instructions, the conversation so far and your latest message. Nothing outside it exists as far as the model is concerned.
What is in it
[ {{hl:system prompt}} ] who the assistant is, what it must and must not do
[ {{hl:conversation}} ] every earlier turn, yours and its own
[ {{hl:attachments}} ] documents, retrieved passages, tool results
[ {{hl:your message}} ] what you just sent
[ {{hl:response so far}} ] the tokens it has produced this turn
{{cm:------------------------------------------------}}
all of it counted in tokens, all of it re-read on every single pass
The sentence that changes how people use these tools
The model retains nothing between requests. Every time you send a message, the entire conversation is submitted again from the beginning. The continuity you experience is produced by resubmission, not by recall.
· Why it is not memory
A desk, not a mind
The distinction has practical consequences worth being precise about.
What follows
Start a new conversation and the model knows nothing about the previous one. Nothing was stored, because storage was never involved.
Products that appear to remember you across sessions are keeping notes outside the model and pasting relevant ones back into the context at the start of each new conversation. That is a feature of the software around the model, and it is why such features can be switched off, edited and deleted. Model parameters cannot be.
And when a conversation gets long enough to exceed the window, something has to give. Systems typically drop or summarize the oldest material, which is the usual explanation for a model losing the thread of something you established early on.
· Why long context is expensive
The cost does not rise in a straight line
Doubling the context does more than double the work in the attention step.
The arithmetic
In the standard mechanism every token is compared against every token it is allowed to see, so the number of comparisons rises with the square of the length. Twice the tokens is four times the comparisons.
A great deal of engineering effort goes into softening that curve, and the effort is why very long context became feasible at all.
There is a separate and more practical point. A model having access to a million tokens is not the same as a model using them well. Reliable retrieval from deep inside a very long context is uneven and worth testing rather than assuming.
Step 07 · Tool calling
How a text predictor comes to act
Everything up to here produces text and nothing else. tool calling is the bridge from a system that writes to a system that does things, and understanding the boundary matters more than any other single thing on this page.
In one sentence
The model writes down a request to run something. Separate software decides whether to run it, runs it, and writes the result back into the conversation.
What actually happens
you : what is the weather in Treviso
model : {{hl:{ "tool": "get_weather", "args": { "city": "Treviso" } }}}
{{cm:this is TEXT. the model has done nothing else.}}
harness : recognizes the request, decides to honor it,
calls the weather service, gets 24C and clear
harness : appends the result into the context window
model : runs again, now with the result in front of it,
and writes: it is 24 degrees and clear in Treviso
The boundary
The model never reaches out to anything. It has no network access, no file access and no ability to execute. It emits a structured request, and a program decides whether to honor it.
Every action ever attributed to an AI agent happened on the far side of that line. Which is where the controls belong, and where anyone assessing the risk of one of these systems should be looking.
· How the model knows what tools exist
They are described in the context
There is no registry inside the model. Available tools are described in text, in the context, the same way everything else is.
The mechanism
The harness writes a description of each tool, its arguments and when to use it, and places that in the context alongside the system prompt. The model reads it like any other input.
Which means tool descriptions occupy the context window and compete for the model's attention with everything else in it. A system offered many tools is spending context on descriptions and is choosing among more options, and both of those make the choice less reliable.
· Where this gets risky
Instructions arriving as data
The model reads the context as one stream, and it has no reliable way to tell instructions from you apart from text that arrived inside a document, a web page or a tool result.
The exposure
If a model can read external content and also call tools, then text placed in that content can attempt to direct its behavior. This is a real and current class of vulnerability, and it exists because instruction and data share one channel by design.
There is no clean fix inside the model, for the same reason: the model is reading one sequence of tokens and every part of it is available to influence what comes next. The controls that work sit in the harness, in what tools are exposed, in what is permitted without a human confirming, and in treating anything the model read from the outside world as untrusted.