Every model โ GPT-2, GPT-4, Llama, this tiny one โ produces the same kind of object: a distribution over next tokens. The difference between "creative writing assistant" and "deterministic calculator" is not the model. It's the decoding. Same weights, different knobs, different behavior. That's why this lesson matters: it's the interface between the model's knowledge and the text you actually see.
Sampling & decoding
The model doesn't write sentences โ it outputs a probability distribution over the next token, one step at a time. Everything that makes the output feel like writing happens after the model: how you pick a token from that distribution, how you handle repetition, how adventurous you let it be. This lesson is about that choice. It's the last piece of the inference loop, and it's where the same checkpoint can produce wildly different text depending on a few numbers.
The model
Same checkpoint, same weights, same forward pass. Nothing in model.py changes. This lesson only touches infer.py's sample() function.
The choice
Temperature, top-k, top-p, repetition penalty. Four knobs, each controlling a different aspect of how you draw from the distribution.
Almost nothing
These are 10โ30 lines each and run in microseconds. The cost is deciding what you want โ coherence vs. variety, safety vs. surprise.
The model is a distribution โ everything else is a choice
Here is the entire output of one forward pass, in the form you actually work with:
softmax(logits) โ probs # positive, sums to 1: P(next token | prefix)
That's the model's job done. It has told you, for every vocabulary item, how likely it is to come next. Everything after this โ which token to actually pick โ is a decision you make, not the model. The model doesn't have an opinion about whether to be creative or safe, repetitive or varied. That's your job, via decoding.
The four decoding strategies in this lesson are four answers to the same question: given this distribution, how do I choose one token?
Try it: shape a distribution
A toy distribution over five tokens. Drag the sliders to see how temperature, top-k, and top-p each reshape the probabilities. The bars show the final probability after all three are applied.
Temperature โ how flat or sharp the distribution is
Temperature divides the logits before softmax. That's it โ one line. But the effect is dramatic: it controls how much the model's confidence is respected.
probs = softmax(scaled_logits) T < 1 โ sharper # confident tokens dominate; output is more predictable T = 1 โ unchanged # the model's native distribution T > 1 โ flatter # low-probability tokens get more chance; output is more varied T โ 0 โ argmax # deterministic: always pick the top token
def sample_with_temperature(logits, temperature=1.0):
if temperature <= 0:
# deterministic: argmax (greedy)
return max(range(len(logits)), key=lambda i: logits[i].data)
scaled = [v / temperature for v in logits]
probs = softmax(scaled)
r = random.random()
acc = 0.0
for i, p in enumerate(probs):
acc += p.data
if r <= acc:
return i
return len(probs) - 1
Why dividing logits flattens the distribution: softmax is exp(x_i) / ฮฃ exp(x_j). If you divide all x_i by T > 1, the differences between logits shrink, so exp compresses the gaps between probabilities. If T < 1, differences grow, and the top token's probability dominates.
Why T = 0 is special-cased: dividing by zero is undefined, and the limit is argmax anyway. So T = 0 means "always pick the highest-probability token" โ greedy decoding. Useful for deterministic tasks (code, math), terrible for creative writing.
| T | Behavior | Good for | Bad for |
|---|---|---|---|
0.0 | Greedy (argmax) โ always top token | Code, math, factual Q&A | Creative writing (repetitive, dull) |
0.3 | Very confident โ top token almost always wins | Precise tasks, summarization | Anything needing variety |
0.7 | Fairly confident, some variety | Most chat, general use | โ |
1.0 | The model's native distribution | Research, baselines | โ |
1.3 | More varied, occasional surprises | Brainstorming, creative writing | Factual tasks (hallucination risk) |
2.0 | Very flat โ low-prob tokens common | Poetry, experimental | Most things (incoherent) |
High temperature makes the model more likely to pick low-probability tokens โ but it doesn't prevent it from picking absurd ones (probability 1e-6). The long tail of the distribution is still there, just slightly amplified. That's why you combine temperature with top-k or top-p: temperature controls how much you trust the model's ranking, top-k/top-p controls how far down the ranking you're allowed to go.
Top-k โ keep only the k most likely tokens
Top-k is the simplest way to prevent the model from picking absurd tokens: keep only the k most probable tokens, set everything else to zero, renormalize, and sample from the rest.
2. keep the top k
3. set all others to 0
4. renormalize so the kept probabilities sum to 1
5. sample
def sample_top_k(logits, k, temperature=1.0):
if k <= 0:
return sample_with_temperature(logits, temperature)
scaled = [v / temperature for v in logits]
probs = softmax(scaled)
# indices of the k highest probabilities
idx = sorted(range(len(probs)), key=lambda i: probs[i].data, reverse=True)[:k]
keep = {i: probs[i].data for i in idx}
z = sum(keep.values())
r = random.random() * z
acc = 0.0
for i, p in keep.items():
acc += p
if r <= acc:
return i
return idx[-1]
k = 1 is greedy decoding (always the top token). k = 50 is the GPT-2 default. k = 0 (or negative) means "off โ sample from the full distribution."
Why renormalize? After zeroing the tail, the kept probabilities no longer sum to 1. Dividing by z makes them a valid distribution again, so the sampling loop works correctly.
The fixed-k problem: k is a count, not a probability mass. If the distribution is very peaked (one token at 0.99), keeping k=50 adds 49 tokens with tiny probabilities โ you've widened the distribution unnecessarily. If the distribution is flat (20 tokens at 0.04 each), keeping k=50 keeps all of them โ you haven't filtered anything. Top-k doesn't adapt to the distribution's shape. That's what top-p fixes.
Top-k is fast (sorting is O(V log V), fine for any vocab size) and simple. It's a good default when you want a hard cap on absurdity and don't care about adaptivity. GPT-2 used it. Modern models mostly use top-p instead, but top-k is still common in production for its predictability: you always sample from exactly k options, which makes the output distribution easier to reason about.
Top-p (nucleus) โ keep the smallest set that sums to p
Top-p (also called nucleus sampling) adapts to the shape of the distribution. Instead of keeping a fixed number of tokens, it keeps the smallest set whose cumulative probability reaches p. If the distribution is peaked, that set is small; if it's flat, the set is large.
2. accumulate probabilities until the sum โฅ p
3. keep that set (the "nucleus")
4. set all others to 0
5. renormalize
6. sample
def sample_top_p(logits, p, temperature=1.0):
if p >= 1.0:
return sample_with_temperature(logits, temperature)
scaled = [v / temperature for v in logits]
probs = softmax(scaled)
# sort descending by probability
order = sorted(range(len(probs)), key=lambda i: probs[i].data, reverse=True)
# smallest prefix whose cumulative probability โฅ p
cum = 0.0
keep = []
for i in order:
keep.append(i)
cum += probs[i].data
if cum >= p:
break
# renormalize and sample
kept = {i: probs[i].data for i in keep}
z = sum(kept.values())
r = random.random() * z
acc = 0.0
for i, pr in kept.items():
acc += pr
if r <= acc:
return i
return keep[-1]
p = 0.9 is the common default โ keep the set that covers 90% of the probability mass. p = 1.0 means "off โ use the full distribution." p = 0.5 is much more focused (only the high-probability core).
Why it adapts: if one token has probability 0.95, the loop stops after the first token โ the nucleus is size 1, and you effectively get greedy decoding. If 50 tokens each have probability 0.02, the loop keeps all 50 โ the nucleus is large, and you get variety. The threshold is fixed, but the number of tokens adapts to how spread out the distribution is.
Top-p vs. top-k: top-p is strictly better in the sense that it handles both peaked and flat distributions well. Top-k can keep too many tokens when the distribution is peaked (adding noise) or too few when it's flat (removing valid options). This is why modern models default to top-p.
For most use cases: temperature=0.7, top_p=0.9, top_k=0 (off), repetition_penalty=1.1. This gives coherent but varied output, avoids absurd tokens, and handles repetition. Adjust from there: raise temperature for creativity, lower it for precision, lower top-p for focus, raise it for variety.
Repetition penalty โ stop the "the the the" loop
Small models (and sometimes large ones) get stuck repeating tokens. The distribution says "the" is slightly more likely than anything else, so it picks "the", then the new distribution also says "the" (because "the the" is a plausible continuation of "the"), and the output degenerates into a loop.
Repetition penalty fixes this by dividing the logits of recently seen tokens by a penalty factor before softmax. A factor of 1.0 is no penalty; 1.2 means "make recently-seen tokens 20% less likely."
if logit[t] > 0: logit[t] /= penalty
else: logit[t] *= penalty
# then softmax as usual
def apply_repetition_penalty(logits, recent_ids, penalty=1.1):
"""Reduce the probability of tokens seen in the recent output."""
if penalty <= 1.0 or not recent_ids:
return logits
seen = set(recent_ids)
out = []
for i, v in enumerate(logits):
if i in seen:
# divide positive logits, multiply negative ones (both push toward 0)
if v.data > 0:
out.append(Value(v.data / penalty))
else:
out.append(Value(v.data * penalty))
else:
out.append(v)
return out
Why the sign branch? Dividing a negative logit by 1.2 makes it less negative (closer to zero), which actually increases its probability โ the opposite of what we want. Multiplying a negative logit by 1.2 makes it more negative, pushing its probability down. So the branch ensures both positive and negative logits are pushed toward lower probability.
How many recent tokens? The recent_ids window is usually the last 64โ256 generated tokens. Too short (e.g., 4) and the model can repeat across a wider gap. Too long and you penalize tokens that are genuinely needed (e.g., "the" is common; penalizing it for the last 1,000 tokens degrades output). A common default is the last 128 tokens, or the full context in modern implementations.
Typical values: 1.0 = off, 1.1 = mild (good default), 1.2 = aggressive, >1.5 = usually too much (output becomes stilted, avoids common words unnaturally).
The trade-off: repetition penalty reduces loops but also makes the model less likely to use a word it should use again. For prompts that genuinely need repetition (e.g., a list of similar items), set penalty=1.0. For free-form generation, 1.1โ1.2 is a safe default.
On this tiny model, repetition happens because the distribution is poorly learned โ the model genuinely thinks "the the the" is a plausible continuation. Repetition penalty masks the symptom, but the root cause is the model's capacity. On GPT-class models, repetition is much rarer because the distribution is much better. So: use repetition penalty to clean up small-model output, but understand that a bigger model would need it less.
Try it: combine the knobs
A toy next-token distribution, shaped by temperature, top-k, top-p, and repetition penalty. This is the same distribution a real model produces; the playground just lets you see how each knob reshapes it before sampling.
Where this leads
This lesson closes the inference loop. You now have:
- The tokenizer (lesson 3): text โ integers.
- The model (lessons 1โ2): integers โ distribution over next token.
- The training loop (lesson 4): how to make the distribution good.
- The decoding strategy (this lesson): how to pick a token from the distribution.
That's the full mechanism, end to end. Everything else is scale, engineering, or alignment โ not new concepts in the core loop.
| Stage | Mechanism | What's new |
|---|---|---|
| Tokens | BPE (lesson 3) | Text โ integers |
| Model | Transformer (lessons 1โ2) | Integers โ distribution |
| Training | Gradient descent (lesson 4) | Distribution gets better |
| Decoding | Sampling (this lesson) | Distribution โ token |
| Scale | PyTorch, GPUs, data | Distribution gets much better |
The next natural step is a PyTorch port: same model, same tokenizer, same training loop, but in a framework that can train 10Mโ100M parameter models on real data. The concepts from lessons 1โ5 transfer directly โ the schedule, the accumulation, the checkpointing, the decoding โ only the implementation speeds up. After that, scale becomes the lesson: you can watch capabilities appear that are absent at 170K parameters.
The source files
Requires: Python 3.9+. No pip install needed. No dependencies beyond the standard library.
Layout: This lesson uses the same six files as lesson 4 โ tokenizer.py, micrograd.py, model.py, data.py, train.py, and infer.py โ plus a trained model.bin and model.tok.json. Only infer.py changes; the other five are identical to lesson 4 and are not repeated here. Run lesson 4's train.py first to produce the checkpoint, then use this lesson's infer.py to sample from it.
# 1. train a model (lesson 4) python train.py corpus/ # 2. sample with the new decoding strategies (this lesson) python infer.py
What changed: infer.py now exposes temperature, top-k, top-p, and repetition penalty as interactive commands, and the sample() function has been factored into composable pieces. The rest of the pipeline is unchanged.
infer.py โ complete (the only changed file)
#!/usr/bin/env python3
"""infer.py โ sample from the subword model with decoding strategies.
New vs. lesson 4:
* temperature (sharpens or flattens the distribution)
* top-k (keep only the k most likely tokens)
* top-p / nucleus (keep the smallest set summing to p)
* repetition penalty (push down recently-seen tokens)
All four compose: temperature first, then penalty, then top-k, then top-p.
"""
import sys, struct, math, random
from pathlib import Path
from tokenizer import BPETokenizer
from micrograd import Value
from model import CTX_LEN, init_model, all_params, forward, softmax
REPETITION_WINDOW = 128 # how many recent tokens to penalize
def load_weights_into(model, flat):
it = iter(flat)
for v in all_params(model):
v.data = next(it)
def load_model_bin(path):
with open(path, "rb") as f:
steps, vocab_size, n_params = struct.unpack("<III", f.read(12))
loss, = struct.unpack("<f", f.read(4))
flat = [struct.unpack("<f", f.read(4))[0] for _ in range(n_params)]
model = init_model(vocab_size)
load_weights_into(model, flat)
return model, steps, vocab_size, loss
# ---------- decoding building blocks ----------
def apply_repetition_penalty(logits, recent_ids, penalty=1.1):
"""Push down logits of tokens seen in the last REPETITION_WINDOW tokens."""
if penalty <= 1.0 or not recent_ids:
return logits
seen = set(recent_ids)
out = []
for i, v in enumerate(logits):
if i in seen:
if v.data > 0:
out.append(Value(v.data / penalty))
else:
out.append(Value(v.data * penalty))
else:
out.append(v)
return out
def _apply_top_k(probs, k):
if k <= 0 or k >= len(probs):
return list(probs)
idx = sorted(range(len(probs)), key=lambda i: probs[i].data, reverse=True)[:k]
keep_set = set(idx)
out = [(probs[i] if i in keep_set else Value(0.0)) for i in range(len(probs))]
return out
def _apply_top_p(probs, p):
if p >= 1.0:
return list(probs)
order = sorted(range(len(probs)), key=lambda i: probs[i].data, reverse=True)
cum = 0.0
keep_set = set()
for i in order:
keep_set.add(i)
cum += probs[i].data
if cum >= p:
break
out = [(probs[i] if i in keep_set else Value(0.0)) for i in range(len(probs))]
return out
def _renormalize(probs):
z = sum(v.data for v in probs)
if z <= 0:
return probs
return [v / z for v in probs]
def _draw(probs):
r = random.random()
acc = 0.0
for i, p in enumerate(probs):
acc += p.data
if r <= acc:
return i
return len(probs) - 1
def sample(logits, temperature=1.0, top_k=None, top_p=1.0, recent_ids=None, repetition_penalty=1.0):
"""One token from a distribution, shaped by four decoding knobs."""
# 0. greedy
if temperature <= 0:
return max(range(len(logits)), key=lambda i: logits[i].data)
# 1. repetition penalty (applied to raw logits, before temperature)
if recent_ids and repetition_penalty > 1.0:
logits = apply_repetition_penalty(logits, recent_ids[-REPETITION_WINDOW:], repetition_penalty)
# 2. temperature
scaled = [v / temperature for v in logits]
probs = softmax(scaled)
# 3. top-k
if top_k:
probs = _apply_top_k(probs, top_k)
# 4. top-p
if top_p < 1.0:
probs = _apply_top_p(probs, top_p)
# 5. renormalize after filtering, then sample
probs = _renormalize(probs)
return _draw(probs)
def _pad_to_ctx(tokens):
if len(tokens) >= CTX_LEN:
return tokens[-CTX_LEN:]
return [32] * (CTX_LEN - len(tokens)) + tokens
def generate(model, tok, prompt, n_tokens=60, temperature=1.0, top_k=None, top_p=1.0, repetition_penalty=1.0):
tokens = tok.encode(prompt)
if not tokens:
tokens = [32]
generated = []
for _ in range(n_tokens):
ctx = _pad_to_ctx(tokens)
logits = forward(ctx, model)
next_id = sample(
logits[-1],
temperature=temperature,
top_k=top_k,
top_p=top_p,
recent_ids=generated, # only newly generated tokens, not the prompt
repetition_penalty=repetition_penalty,
)
tokens.append(next_id)
generated.append(next_id)
return tok.decode(tokens)
def main():
random.seed(0)
model_bin = Path(sys.argv[1] if len(sys.argv) > 1 else "model.bin")
tok_json = Path(sys.argv[2] if len(sys.argv) > 2 else "model.tok.json")
if not model_bin.exists():
raise SystemExit(f"missing {model_bin}; run train.py first")
if not tok_json.exists():
raise SystemExit(f"missing {tok_json}; run train.py first")
model, steps, vocab_size, loss = load_model_bin(model_bin)
tok = BPETokenizer.load(tok_json)
if tok.vocab_size != vocab_size:
raise SystemExit(f"vocab mismatch: model={vocab_size} tokenizer={tok.vocab_size}")
print(f"Loaded: {steps} steps, vocab {vocab_size}, val loss {loss:.4f}")
print("Commands:")
print(" :t <temp> temperature (0=greedy, 0.7=default, 2=chaos)")
print(" :k <top_k> top-k (0=off)")
print(" :p <top_p> top-p (1.0=off)")
print(" :r <penalty> repetition penalty (1.0=off)")
print(" :q quit")
print(" (or type a prompt to generate)")
print()
temp, topk, topp, rep = 1.0, None, 1.0, 1.0
while True:
try:
line = input("> ").strip()
except (EOFError, KeyboardInterrupt):
print()
break
if not line:
continue
if line == ":q":
break
if line.startswith(":t "):
temp = float(line[3:])
print(f"temperature = {temp}")
continue
if line.startswith(":k "):
k = int(line[3:])
topk = k if k > 0 else None
print(f"top_k = {topk}")
continue
if line.startswith(":p "):
topp = float(line[3:])
print(f"top_p = {topp}")
continue
if line.startswith(":r "):
rep = float(line[3:])
print(f"repetition_penalty = {rep}")
continue
out = generate(
model, tok, line,
n_tokens=60,
temperature=temp,
top_k=topk,
top_p=topp,
repetition_penalty=rep,
)
print(out)
print()
if __name__ == "__main__":
main()
Everything is repetitive โ raise :r to 1.1โ1.2, or lower temperature.
Everything is incoherent โ lower temperature (try 0.5), set :p 0.9, and set :r 1.1.
Output never varies between runs โ temperature is 0 (greedy). Raise it.
Top-p does nothing โ make sure :p is set below 1.0. At p=1.0 it's off.
Repetition penalty makes output stilted โ lower it. 1.3+ is usually too aggressive; 1.1 is a safe default.
Output contains garbage tokens โ top-k or top-p isn't filtering enough. Raise top_p strictness (lower p) or set :k 50.
What you just learned
1. The model is a distribution, decoding is the choice. Every forward pass returns a probability distribution over the next token. The model has no opinion about creativity, safety, or repetition โ those are all decisions made in sample(). Same checkpoint, different knobs, different output.
2. Temperature, top-k, top-p, and repetition penalty compose. Temperature shapes the distribution; top-k and top-p filter it; repetition penalty biases it. The recommended default is temperature=0.7, top_p=0.9, top_k=0, repetition_penalty=1.1. Adjust from there based on the task.
3. Top-p is usually better than top-k. Top-k keeps a fixed number of tokens, which is wrong for both peaked and flat distributions. Top-p keeps the smallest set that covers p of the mass, which adapts to the distribution's shape. Modern models default to top-p; top-k is still useful when you want predictability.
You now have the complete inference loop: text โ tokens โ distribution โ sampled token โ repeat โ detokenize. The same loop runs on this 170K-parameter model and on GPT-4; the difference is entirely in how good the distribution is (scale) and how the token is chosen (this lesson). Everything after this โ PyTorch, scale, alignment โ assumes you understand this loop, because it's the loop that turns weights into text.
The next lesson is the PyTorch port: same model, same tokenizer, same training loop, same decoding โ but in a framework that can train 10Mโ100M parameter models on real data, where scale starts to matter and capabilities start to appear. The pure-Python arc ends here; the rest is engineering and scale.