The character model from the previous lesson spends most of its capacity on spelling. Every token has to learn that "q" is usually followed by "u," that "e" is common, that words end in space. The word model skips all of that โ it starts with whole words and spends its capacity on grammar and co-occurrence. Caveat: of the ~153K parameters, ~128K sit in the embedding table and output head, so the Transformer blocks (~25K) are no bigger than before. The gain is that the blocks no longer have to learn spelling.
From characters to words
The previous lesson built a character-level Transformer that predicts one character at a time. This lesson upgrades that same model to predict the next word. The architecture doesn't change โ the Transformer block and optimizer are identical, and the autograd engine needs just one fix. What changes is the tokenizer, the vocabulary size, and the context length. Those three changes ripple through training, loss numbers, and generation.
All the internals
Autograd, attention, LayerNorm, GELU, Adam, the save format, the sampling loop. Only one small fix: backward() becomes iterative (Step 4 notes why).
Three things
Tokenizer (whitespace split instead of per-character), vocabulary (~2,000 words instead of ~105 characters), context (16 words instead of 32 characters).
Bigger and slower
Parameters rise ~33K โ ~153K (bigger embedding table and output head). Training is ~1.6ร slower per step and uses ~2ร the memory, because the softmax now spans 2,001 words.
Why words instead of characters
This model does the same thing as the previous character-level model: given a window of tokens, it predicts the next token. The only difference is what counts as a "token."
| Character model (previous lesson) | Word model (this lesson) | |
|---|---|---|
| Token | one character | one whitespace-delimited word |
| Vocab size | ~105 (all unique chars) | ~2,001 (2,000 common words + <unk>) |
| Context | 32 chars โ 6 words | 16 words โ one sentence |
| Random-guess loss | ln(105) โ 4.65 | ln(2001) โ 7.60 |
| Trained loss | ~2.7 (ppl โ14) | lower than the ~5.5 unigram baseline (see Step 5) |
| Training time (300 steps, pure Python) | ~16โ19 min | ~26โ32 min |
| Output | The t atharexthe te are is | The code is a of the can be |
Three things jump out from this comparison:
- The loss numbers look worse but aren't. A higher loss means the task is harder per prediction โ picking 1 of 2,001 words is harder than picking 1 of 105 characters. Perplexity is the number of equally likely choices the model is effectively torn between, so a word model at perplexity โ60 is doing real work out of 2,001 options.
- Training is slower, not faster. Fewer tokens per window does not mean less work. The output layer now scores 2,001 words at every position, and the softmax over them dominates: measured per window, the word model builds ~322K autograd nodes vs ~228K for the character model (about 60% of them in the softmax), and a step takes ~1.6ร longer.
- The output is more readable. The word model outputs real words joined by spaces, so even when it's incoherent ("The code is a of the can be") you can read it. The character model outputs strings like "atharexthe" that take effort to decode.
The tokenizer โ the actual new code
In the previous lesson, the tokenizer was implicit: each character was already a token, so the "tokenizer" was just [stoi[c] for c in text]. In this lesson, the tokenizer becomes real code. It's the only genuinely new code in the upgrade; everything else is constants and prose.
The tokenizer has two directions: text โ token ids (used in training), and token ids โ text (used in generation).
from collections import Counter
def build_vocab(text, size):
"""Return the `size` most common words, in frequency order."""
counts = Counter(text.split())
return [w for w, _ in counts.most_common(size)]
def tokenize(text, stoi):
"""Split on whitespace; map each word to its index. Unknown โ <unk>."""
unk = len(stoi) # <unk> is always the last index
return [stoi.get(w, unk) for w in text.split()]
text.split() is the whole tokenizer. Python's default split() handles any run of whitespace (spaces, tabs, newlines) and returns a list of words. This is intentionally crude: real systems use BPE or SentencePiece, which break rare words into frequent subword pieces. But for a 2,000-word vocabulary on a modest corpus, whitespace splitting is enough to demonstrate the principle.
Counter(text.split()) gives you word frequencies in one line. most_common(size) returns the top-N as (word, count) pairs. The vocabulary order matters โ it determines which index each word gets โ but any consistent order works.
The <unk> slot is reserved as the last index. Any word not in the top 2,000 collapses to this single token. That means the model learns one embedding for "some rare word I've never seen," which is a coarse but workable signal: it can learn that <unk> often appears after "the" or before a period, even without knowing which specific word it is.
Compare to the previous lesson: there, chars = sorted(set(text)) gave a vocabulary automatically โ every character that appears anywhere must be representable. Here, the vocabulary is chosen, which is why the next step is about vocabulary size.
"the," and "the" are different tokens โ the comma makes them distinct. This is the biggest weakness of naive whitespace tokenization: the vocabulary fills up with word, word,, word. variants. On Python's reference docs this is costly: the top 2,000 whitespace tokens cover only ~84% of the text. A real tokenizer would split punctuation off. The regex re.findall(r"\w+|[^\w\s]", text) does that, but you must also change prompt tokenization and the detokenizer, since " ".join() would put spaces before punctuation.
Vocabulary size is the key knob
In the previous character-level model, the vocabulary was determined by the corpus โ every character that appears anywhere must be in the vocabulary, or it can't be represented. In this word-level model, the vocabulary size is a choice. That choice is the most important new decision in the upgrade.
| Vocab size | Params (approx) | Coverage | Effect |
|---|---|---|---|
| 500 | ~57K | ~67% of tokens | Many <unk>; loss stays high |
| 2,000 | ~153K | ~84% of tokens | The reference setting |
| 5,000 | ~345K | ~94% of tokens | Bigger table, slower; diminishing returns |
| all (~9.2K) | ~610K | 100% | Too big for pure Python to train well |
Coverage was measured on a 467K-character corpus built from Python's reference docs (pydoc_data.topics) with whitespace splitting; other corpora will differ. The trade-off is stark: the embedding table and output head scale linearly with vocabulary size. At 2,000 words those two matrices alone account for 2 ร 32 ร 2001 โ 128K of the ~153K parameters. The Transformer blocks only add ~25K. Parameters follow 64 ร vocab + 25,408, so going from 2,000 to 5,000 words more than doubles the model (153K โ 345K), and the softmax cost grows with it.
tok_embed = 2001 ร 32 = 64,032 params
W_head = 2001 ร 32 = 64,032 params
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
embeddings = 128,064 params โ ~5ร the rest of the model
blocks = 24,832 params (LayerNorms, W_qkv, W_out, FFNs)
pos_embed + final LayerNorm = 576 params
total = 153,472 params
On the reference corpus the top 2,000 whitespace tokens cover ~84% of text, so about one token in six becomes <unk>. Going to 5,000 reaches ~94% but more than doubles parameters and slows every step. 2,000 is a compromise for pure-Python training time, not a point where coverage is 'enough'.
A 50,000-word English vocabulary would need a 50K ร 768 embedding table โ 38M parameters just for the input embeddings, before any actual model. BPE and SentencePiece exist to solve exactly this: they build a vocabulary of ~30Kโ50K subword pieces (common words, common word fragments, punctuation) that covers nearly 100% of any text at a manageable vocab size. The next stage of the scaling path below mentions this.
New shape โ context length drops
Only one constant changed besides the vocab: CTX_LEN goes from 32 to 16. Everything else โ D_MODEL, N_LAYERS, N_HEADS, D_FF โ stays exactly as it was in the previous lesson.
# previous lesson (character model)
CTX_LEN = 32 # 32 characters โ 6 words
# this lesson (word model)
CTX_LEN = 16 # 16 words โ one sentence
# everything else unchanged from the previous lesson
D_MODEL = 32
N_LAYERS = 2
N_HEADS = 2
D_FF = 128
Why fewer tokens for more context? Because a token now carries roughly 5โ7 characters of text instead of 1 (6.8 on the reference corpus, counting the space). Sixteen words is roughly a full sentence of context โ enough for the model to see "The cat sat on the ___" and predict "mat." Thirty-two characters, as in the previous lesson, is only about five or six words.
Only the attention part scales with the square of the context; the FFN, output head and softmax scale linearly. Measured here, doubling CTX_LEN from 16 to 32 raised the time per window from ~1.6 s to ~3.1 s, about 1.9ร, not 4ร. Sixteen words is the compromise where you get sentence-level context without paying a fortune in wall-clock time.
Parameter count, itemized
Here's the full breakdown of where the ~153K parameters go, compared to the previous character model:
| Component | Character model (previous) | Word model (this lesson) |
|---|---|---|
tok_embed | 105 ร 32 = 3,360 | 2001 ร 32 = 64,032 |
pos_embed | 32 ร 32 = 1,024 | 16 ร 32 = 512 |
| 2 ร LayerNorm | ~256 | ~256 |
2 ร W_qkv | 2 ร 3 ร 32ยฒ = 6,144 | 6,144 |
2 ร W_out | 2 ร 32ยฒ = 2,048 | 2,048 |
| 2 ร FFN | 2 ร (32ร128 + 128ร32) = 16,384 | 16,384 |
| Final LayerNorm | 64 | 64 |
W_head | 32 ร 105 = 3,360 | 32 ร 2001 = 64,032 |
| Total | ~32,640 | 153,472 |
Notice: the Transformer blocks are identical in both columns. Almost the entire parameter increase comes from the embedding table and output head, both of which grew because the vocabulary grew. This is the clean lesson of the upgrade: changing the tokenizer changes the vocabulary, which changes the parameter count, without touching the model architecture.
Training โ what actually changes
The training loop code is identical to the previous lesson. What changes is the data that flows through it, and therefore the loss numbers you'll see.
What you already have from the previous lesson (unchanged)
- Sample โ forward โ backward โ Adam update, four operations per step
- Adam with bias correction, written out explicitly
p.grad = 0.0after each updategc.disable()during training- 300 steps, batch size 4, learning rate 3e-3
- Not unchanged:
backward()inmicrograd.pyis now iterative (see the source section)
The only change inside the loop is that data is now a list of word ids instead of character ids. The slicing logic x = data[j : j+CTX_LEN] is identical.
The loss curve you'll see
Format of the training log, with the numbers I could verify. A 4-step run on a 467K-character corpus built from Python's reference docs gave Params: 153,472, 68,827 tokens, loss 7.60 at step 1 (= ln 2001), ~5.1 s per step, and ~1.5 GB peak memory. A full 300-step run therefore takes roughly 26 minutes on a single core. The loss values below come from the original author's run and I could not reproduce them here, so expect different numbers:
Vocab: 2001 Tokens: 68,827
Params: 153,472
Training for 300 steps...
step 1 loss 7.6035
step 50 loss 6.0021
step 100 loss 5.2047
step 150 loss 4.7535
step 200 loss 4.5486
step 300 loss 4.1463
Saved: model.bin (avg loss over last 50 steps: 4.1463)
Done in ~1,500s (measured ~5.1 s/step in a 4-step run)
Each printed loss is the average of the last 50 steps, not a single step, and it is training loss on random windows (no held-out set). Compare to the character version from the previous lesson (I measured ~3.2 s/step and ~680 MB for it, so ~16 min total, consistent with the 19 min below):
Vocab: 105 Tokens: 467,243
Params: 32,640
step 1 loss 4.6803
step 50 loss 3.6219
step 100 loss 3.0556
step 150 loss 2.8065
step 200 loss 2.7181
step 300 loss 2.6631
Done in 1143.2s
Character (previous lesson): 4.68 โ 2.66 (drop of 2.02). Random-guess baseline 4.65. Perplexity 105 โ 14.
Word (this lesson): 7.60 โ 4.15 (drop of 3.45). Random-guess baseline 7.60. Perplexity 2001 โ 63.
Per-token losses are not comparable across tokenizers, because a word is worth ~5โ7 characters. A fairer check is the unigram baseline: always predicting word frequencies gives ~5.5 nats on the reference corpus, so a word model below that has learned context. Note also that <unk> hides the spelling of rare words, which flatters the word model. Wall-clock, the word model is ~1.6ร slower per step, for the softmax reasons in Step 1.
The word model's data is ~68.8K tokens on the reference corpus (down from ~467K characters). Each step samples BATCH_SIZE ร CTX_LEN = 64 tokens, so 300 steps touch 19,200 tokens โ about 28% of the corpus. The character version saw 38,400 of 467,243 โ about 8%. In text terms, 19,200 words is ~130K characters, about 3.4ร more text per run than the character model sees, so the same 300 steps cover far more of the corpus.
Generation โ words are joined, not concatenated
The generation loop is the same as in the previous lesson: forward, sample, append, repeat. Two changes: the prompt is tokenized by splitting on whitespace, and the output is joined with spaces.
# ---- tokenize the prompt ----
# previous lesson: [stoi.get(c, unk) for c in prompt]
tokens = [stoi.get(w, unk) for w in prompt.split()] or [unk]
# ---- ... generation loop identical to the previous lesson ... ----
# ---- detokenize the output ----
# previous lesson: "".join(itos.get(t, "<unk>") for t in tokens)
return " ".join(itos.get(t, "<unk>") for t in tokens)
That's the entire diff. The .split() on input and the " ".join() on output are the only tokenizer-aware lines in generation.
Real output from the trained model
Same corpus, same 300 steps, but word tokenization. Temperature 0.5, top-k 10:
> :t 0.5
> :k 10
> The
The code is a of the can be a the a to be is not a that the to be the a to be a in the a of the a be the a is a in the to be a
What's happening in this output: the model shows only weak local structure. "the code is a ___" is a real English construction. "a of the" is not, but it's a common failure mode: the model has learned that "a" often precedes a noun phrase, and "of the" is a frequent bigram, and it can't distinguish "a piece of" from "a of." At this size and training length, expect mostly frequent function words and local patterns, with no global coherence. This sample is from the author's run; mine (an untrained 4-step model) was gibberish, as expected.
Compare to the previous lesson's character output: The t atharexthe te are is. That's harder to read because the model is still learning English spelling. The word model has already skipped the spelling stage, so it produces immediately recognizable words โ just not sensible sentences.
Lower temperature (:t 0.1) โ the model almost always picks its single most likely next word. Output becomes repetitive but you can see what the model actually "wants" to say.
Raise it (:t 1.5) โ much more random, output degrades to noise.
No top-k (:k 0) โ sample from all 2,001 words every step. Much more random than top-k 40.
Long prompt โ type 20 words instead of one. The model only sees the last 16, so anything earlier is ignored, and during generation the start of your prompt scrolls out of view.
The trade-offs, made explicit
Going word-level isn't strictly better than the character-level model from the previous lesson โ it's a different point on the same trade-off curve. Here's the full comparison.
| Property | Character (previous) | Word (this lesson) | Who wins |
|---|---|---|---|
| Parameters (same architecture) | ~33K | ~153K | Character (smaller) |
| Training time | ~16โ19 min | ~26โ32 min | Character (cheaper softmax) |
| Random-guess loss | 4.65 | 7.60 | โ |
| Trained loss | 2.66 | not comparable per token | โ |
| Perplexity | โ14 | see Step 5 | โ |
| Context window | 32 chars โ 6 words | 16 words โ one sentence | Word (longer semantic context) |
| Output readability | Character fragments | Real words | Word |
| Handles typos/novel words | Yes (spells them) | No (<unk>) | Character |
| Punctuation handling | Native | Attached to words | Character |
| Vocab is a choice | No (fixed by corpus) | Yes (this is the knob) | Word (more flexible) |
When to pick which
- Character if you need to handle arbitrary text without an unknown-token problem โ code, math, mixed-language input, or any corpus where you can't afford to lose rare words.
- Word if you want a model that produces readable output at a small parameter budget, and your corpus is normal English prose where common words dominate.
- Neither if you're doing this for real โ subword tokenization (BPE, SentencePiece) gets you the readability of words and the coverage of characters, at the cost of one more concept to learn.
In the previous character-level lesson, tokenization was invisible โ characters are the obvious unit, and there was nothing to decide. In this word-level lesson, tokenization becomes a design choice: vocabulary size, context length, how to handle unknown words, whether to split punctuation. Those choices propagate through parameter count, training time, and output quality. This is exactly why real language models spend so much engineering effort on tokenizers โ it's the first decision in the pipeline, and it constrains everything downstream.
Where this leads
The word model is one step past the character model, but it's still not how real systems tokenize. The natural next step is subword tokenization โ the actual technique used by GPT, Llama, and every other modern language model.
The subword idea in one paragraph
Start with a vocabulary of single characters. Repeatedly find the most frequent adjacent pair of tokens in your corpus, merge them into a new token, and add it to the vocabulary. Do this ~30,000 times and you get a vocabulary that includes all characters (or bytes, as GPT-2 does, so nothing is ever <unk>), all common words (like "the" and "of"), and all common word fragments (like "ing" and "un-"). This is Byte Pair Encoding (BPE), and it's ~100 lines to implement from scratch. It solves all three problems at once: no unknown tokens, reasonable vocabulary size, and readable output.
The scaling path, updated
| Stage | Tokenizer | Vocab | Context | Params |
|---|---|---|---|---|
| Character (previous lesson) | per-character | ~105 | 32 chars | ~33K |
| Word (this lesson) | whitespace split | ~2,001 | 16 words | ~153K |
| BPE from scratch | byte pair encoding | ~2,000 | 32 subwords | ~154K |
| GPT-2 small | BPE | 50,257 | 1024 subwords | 124M |
Each step in that table is a real, doable upgrade from the one above it. The word model is the natural bridge between the two extremes โ small enough to understand in full, but already touching the real tokenization decision that every larger model has to make.
The full source
Requires: Python 3.9+. No pip install needed. No dependencies beyond the standard library.
Layout: all five files in one folder, plus a corpus.txt.
mkdir docs && cp some_text.md docs/ python build_corpus.py ./docs corpus.txt python train.py corpus.txt model.bin python infer.py
Expected: 153,472 parameters once the corpus has 2,000+ distinct words. In my test container (1 core) training took ~5 s per step (~26 min for 300 steps), peak memory was ~1.5 GB (make sure you have 2+ GB free), and generation took ~0.18 s per word. Tested on Python 3.12.
micrograd.py โ one fix: iterative backward()
"""
micrograd.py โ a tiny autograd engine.
Every operation returns a Value that knows how to push its gradient
back to the Values that produced it. Call .backward() on any Value.
Usage:
a = Value(2.0)
b = Value(3.0)
c = a * b + a
c.backward()
print(a.grad) # 4.0 (dc/da = b + 1)
print(b.grad) # 2.0 (dc/db = a)
"""
import math
class Value:
"""A number that remembers how it was made."""
def __init__(self, data, children=(), op=""):
self.data = data
self.grad = 0.0 # d(output)/d(this value)
self._prev = children # values that produced this one
self._op = op
self._backward = lambda: None
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), "+")
def _backward():
self.grad += out.grad # d(a+b)/da = 1
other.grad += out.grad # d(a+b)/db = 1
out._backward = _backward
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), "*")
def _backward():
self.grad += other.data * out.grad # d(ab)/da = b
other.grad += self.data * out.grad # d(ab)/db = a
out._backward = _backward
return out
def __pow__(self, n):
out = Value(self.data ** n, (self,), f"**{n}")
def _backward():
self.grad += n * (self.data ** (n - 1)) * out.grad
out._backward = _backward
return out
def exp(self):
out = Value(math.exp(self.data), (self,), "exp")
def _backward():
self.grad += out.data * out.grad # d(e^x)/dx = e^x
out._backward = _backward
return out
def log(self):
x = max(self.data, 1e-30) # guard against log(0)
out = Value(math.log(x), (self,), "log")
def _backward():
self.grad += (1.0 / x) * out.grad
out._backward = _backward
return out
def tanh(self):
out = Value(math.tanh(self.data), (self,), "tanh")
def _backward():
self.grad += (1.0 - out.data ** 2) * out.grad
out._backward = _backward
return out
def backward(self):
"""Compute gradients for every Value that produced this one."""
topo, visited = [], set()
stack = [(self, False)] # iterative DFS: a recursive version would hit
while stack: # Python's recursion limit on long chains
v, expanded = stack.pop() # (softmax over 2,001 words is a 2,001-deep sum)
if expanded:
topo.append(v)
continue
if v in visited:
continue
visited.add(v)
stack.append((v, True)) # revisit v after all its children
for child in v._prev:
if child not in visited:
stack.append((child, False))
self.grad = 1.0 # d(output)/d(output) = 1
for v in reversed(topo):
v._backward()
v._backward = lambda: None # free the graph as we go: breaking the
v._prev = () # closure cycles lets Python reclaim memory
topo.clear(); visited.clear()
# helper operators so `2 * v`, `v - 1` and `v / 2` work
def __radd__(self, other): return self + other
def __rmul__(self, other): return self * other
def __neg__(self): return self * -1
def __sub__(self, other): return self + (-other)
def __truediv__(self, other):
other = other if isinstance(other, Value) else Value(other)
return self * (other ** -1)
Value.__radd__ = __radd__
Value.__rmul__ = __rmul__
Value.__neg__ = __neg__
Value.__sub__ = __sub__
Value.__truediv__ = __truediv__
def dot(xs, ws):
"""Dot product as ONE graph node (instead of 2n tiny + and * nodes).
This single trick makes the model ~100x cheaper in memory and time."""
out = Value(sum(x.data * w.data for x, w in zip(xs, ws)),
tuple(xs) + tuple(ws), "dot")
def _backward():
g = out.grad
for x, w in zip(xs, ws):
x.grad += w.data * g # d(xยทw)/dx = w
w.grad += x.data * g # d(xยทw)/dw = x
out._backward = _backward
return out
model.py โ one constant changed, plus a 2-byte word-length prefix
"""model.py โ model definition, primitives, forward pass, checkpoints."""
import math, random, struct
from micrograd import Value, dot
# ---- the shape (small enough to train in pure Python) ----
CTX_LEN = 16 # words visible at once (was 32 chars in the previous lesson)
D_MODEL = 32 # width of each word's vector
N_LAYERS = 2 # Transformer blocks, stacked
N_HEADS = 2 # attention heads per block
D_FF = 128 # FFN inner width (4x D_MODEL)
HEAD_DIM = D_MODEL // N_HEADS
# ---- initialization ----
def init_matrix(rows, cols, scale=0.02):
return [[Value(random.gauss(0, scale)) for _ in range(cols)] for _ in range(rows)]
def init_model(vocab_size):
p = {}
p["tok_embed"] = init_matrix(vocab_size, D_MODEL, scale=0.1)
p["pos_embed"] = init_matrix(CTX_LEN, D_MODEL, scale=0.1)
for i in range(N_LAYERS):
p[f"ln1_g_{i}"] = [Value(1.0) for _ in range(D_MODEL)]
p[f"ln1_b_{i}"] = [Value(0.0) for _ in range(D_MODEL)]
p[f"W_qkv_{i}"] = init_matrix(D_MODEL, 3 * D_MODEL)
p[f"W_out_{i}"] = init_matrix(D_MODEL, D_MODEL)
p[f"ln2_g_{i}"] = [Value(1.0) for _ in range(D_MODEL)]
p[f"ln2_b_{i}"] = [Value(0.0) for _ in range(D_MODEL)]
p[f"W_ff1_{i}"] = init_matrix(D_MODEL, D_FF)
p[f"W_ff2_{i}"] = init_matrix(D_FF, D_MODEL)
p["ln_f_g"] = [Value(1.0) for _ in range(D_MODEL)]
p["ln_f_b"] = [Value(0.0) for _ in range(D_MODEL)]
p["W_head"] = init_matrix(D_MODEL, vocab_size)
return p
def all_params(params):
for v in params.values():
for row in v:
if isinstance(row, list):
yield from row
else:
yield row
# ---- primitives ----
def matmul(X, W):
"""X: list of rows, W: list of rows. Returns X @ W."""
cols = list(zip(*W)) # columns of W
return [[dot(x_row, col) for col in cols] for x_row in X]
def add_rows(X, Y):
return [[a + b for a, b in zip(rx, ry)] for rx, ry in zip(X, Y)]
def softmax(row):
m = max(v.data for v in row) # subtract max for stability
shifted = [v - m for v in row]
exps = [v.exp() for v in shifted]
total = sum(exps, Value(0.0))
return [v / total for v in exps]
def layernorm(row, gamma, beta, eps=1e-5):
n = len(row)
mean = sum(row, Value(0.0)) / n
diffs = [v - mean for v in row]
var = sum((d * d for d in diffs), Value(0.0)) / n
denom = (var + eps) ** 0.5
norm = [d / denom for d in diffs]
return [g * x + b for g, x, b in zip(gamma, norm, beta)]
def gelu_approx(x):
c = 0.7978845608 # sqrt(2/pi)
return 0.5 * x * (1.0 + ((x + 0.044715 * x * x * x) * c).tanh())
# ---- attention + block ----
def attention(h, W_qkv, W_out):
T = len(h)
qkv = matmul(h, W_qkv) # (T, 3 * D_MODEL)
scale = 1.0 / math.sqrt(HEAD_DIM)
out = [[] for _ in range(T)]
for head in range(N_HEADS):
lo = head * HEAD_DIM
q = [r[lo : lo + HEAD_DIM] for r in qkv]
k = [r[D_MODEL + lo : D_MODEL + lo + HEAD_DIM] for r in qkv]
v = [r[2 * D_MODEL + lo : 2 * D_MODEL + lo + HEAD_DIM] for r in qkv]
for t in range(T):
# causal mask: position t only looks at positions 0..t
scores = [dot(q[t], k[s]) * scale for s in range(t + 1)]
weights = softmax(scores)
for d in range(HEAD_DIM):
out[t].append(dot(weights, [v[s][d] for s in range(t + 1)]))
return matmul(out, W_out) # heads are concatenated in `out`
def block(h, p, i):
# 1. attention sublayer
a = [layernorm(row, p[f"ln1_g_{i}"], p[f"ln1_b_{i}"]) for row in h]
h = add_rows(h, attention(a, p[f"W_qkv_{i}"], p[f"W_out_{i}"]))
# 2. feed-forward sublayer
a = [layernorm(row, p[f"ln2_g_{i}"], p[f"ln2_b_{i}"]) for row in h]
ff = matmul(a, p[f"W_ff1_{i}"]) # (T, D_FF)
ff = [[gelu_approx(x) for x in row] for row in ff]
ff = matmul(ff, p[f"W_ff2_{i}"]) # (T, D_MODEL)
return add_rows(h, ff)
def forward(tokens, p):
h = [[p["tok_embed"][tok][d] + p["pos_embed"][t][d] for d in range(D_MODEL)]
for t, tok in enumerate(tokens)]
for i in range(N_LAYERS):
h = block(h, p, i)
h = [layernorm(row, p["ln_f_g"], p["ln_f_b"]) for row in h]
return matmul(h, p["W_head"]) # (T, vocab)
# ---- checkpoints ----
SHAPE = (CTX_LEN, D_MODEL, N_LAYERS, N_HEADS, D_FF)
def save_model(path, params, vocab, step, loss):
with open(path, "wb") as f:
f.write(struct.pack("<5I", *SHAPE)) # so we can detect a mismatch
f.write(struct.pack("<I", len(vocab)))
for word in vocab:
b = word.encode("utf-8") # words can be long (URLs, identifiers)
f.write(struct.pack("<H", len(b)) + b) # 2-byte length prefix, then bytes
f.write(struct.pack("<If", step, loss))
for p in all_params(params):
f.write(struct.pack("<f", p.data))
def load_model(path):
with open(path, "rb") as f:
if struct.unpack("<5I", f.read(20)) != SHAPE:
raise SystemExit("model.bin was trained with different constants")
n_vocab = struct.unpack("<I", f.read(4))[0]
vocab = []
for _ in range(n_vocab):
n = struct.unpack("<H", f.read(2))[0]
vocab.append(f.read(n).decode("utf-8"))
step, loss = struct.unpack("<If", f.read(8))
params = init_model(n_vocab + 1) # +1 for <unk>
for p in all_params(params):
p.data = struct.unpack("<f", f.read(4))[0]
return params, vocab, step, loss
train.py โ tokenizer and vocabulary added
#!/usr/bin/env python3
"""
train.py โ train the tiny model in pure Python.
Usage:
python train.py corpus.txt model.bin
"""
import sys, time, random, gc
from pathlib import Path
from collections import Counter
from micrograd import Value
from model import CTX_LEN, init_model, all_params, forward, softmax, save_model
BATCH_SIZE = 4
STEPS = 300
LR = 3e-3
VOCAB_SIZE = 2000 # most common words; rest become <unk>
def build_vocab(text, size):
"""Return the `size` most common words, in frequency order."""
counts = Counter(text.split())
return [w for w, _ in counts.most_common(size)]
def tokenize(text, stoi):
"""Split on whitespace; map each word to its index. Unknown โ <unk>."""
unk = len(stoi) # <unk> is the last index
return [stoi.get(w, unk) for w in text.split()]
def compute_loss(logits, targets):
"""Cross-entropy, averaged over positions. logits: (T, vocab), targets: (T,)."""
total = Value(0.0)
for t in range(len(logits)):
probs = softmax(logits[t])
total = total - probs[targets[t]].log()
return total / len(logits)
def main():
random.seed(0)
gc.disable() # the graph is freed by refcount; the cycle collector only slows us down
corpus_path = Path(sys.argv[1] if len(sys.argv) > 1 else "corpus.txt")
out_path = Path(sys.argv[2] if len(sys.argv) > 2 else "model.bin")
text = corpus_path.read_text(encoding="utf-8")
vocab = build_vocab(text, VOCAB_SIZE)
stoi = {w: i for i, w in enumerate(vocab)}
data = tokenize(text, stoi)
if len(data) <= CTX_LEN + 1:
raise SystemExit(f"Corpus too short: {len(data)} tokens")
params = init_model(len(vocab) + 1) # +1 for <unk>
plist = list(all_params(params))
print(f"Vocab: {len(vocab) + 1} Tokens: {len(data):,}")
print(f"Params: {len(plist):,}")
m = [0.0] * len(plist) # Adam: running mean of gradients
v = [0.0] * len(plist) # Adam: running mean of squared gradients
b1, b2, eps = 0.9, 0.999, 1e-8
t0, recent = time.time(), []
print(f"Training for {STEPS} steps...")
for step in range(1, STEPS + 1):
# 1. sample a batch of windows
loss = Value(0.0)
for _ in range(BATCH_SIZE):
j = random.randint(0, len(data) - CTX_LEN - 1)
x = data[j : j + CTX_LEN]
y = data[j + 1 : j + CTX_LEN + 1]
# 2. forward pass
loss = loss + compute_loss(forward(x, params), y)
loss = loss / BATCH_SIZE
# 3. backward pass
loss.backward()
# 4. Adam update
for i, p in enumerate(plist):
m[i] = b1 * m[i] + (1 - b1) * p.grad
v[i] = b2 * v[i] + (1 - b2) * p.grad ** 2
m_hat = m[i] / (1 - b1 ** step)
v_hat = v[i] / (1 - b2 ** step)
p.data -= LR * m_hat / (v_hat ** 0.5 + eps)
p.grad = 0.0
recent.append(loss.data)
if step == 1 or step % 50 == 0:
avg = sum(recent[-50:]) / len(recent[-50:])
print(f"step {step:4d} loss {avg:.4f}")
final = sum(recent[-50:]) / len(recent[-50:])
save_model(out_path, params, vocab, STEPS, final)
print(f"Saved: {out_path} (avg loss over last 50 steps: {final:.4f})")
print(f"Done in {time.time() - t0:.1f}s")
if __name__ == "__main__":
main()
infer.py โ two lines changed
#!/usr/bin/env python3
"""infer.py โ generate text from a trained model."""
import sys, random, gc
from pathlib import Path
from micrograd import Value
from model import CTX_LEN, forward, softmax, load_model
def generate(params, vocab, prompt, n_tokens=50, temperature=0.8, top_k=40):
stoi = {w: i for i, w in enumerate(vocab)}
itos = {i: w for i, w in enumerate(vocab)}
unk = len(vocab)
# tokenize prompt by splitting on whitespace (same as training)
tokens = [stoi.get(w, unk) for w in prompt.split()] or [unk]
for _ in range(n_tokens):
# 1. forward pass on the last CTX_LEN tokens
gc.disable() # a forward pass allocates a huge graph;
logits = forward(tokens[-CTX_LEN:], params) # collecting mid-way is very slow
nxt = [Value(v.data) for v in logits[-1]] # keep only the numbers we need
del logits
gc.enable(); gc.collect() # nothing calls backward(), so free it here
# 2. temperature scaling
nxt = [v * (1.0 / max(temperature, 1e-6)) for v in nxt]
# 3. top-k filtering: keep only the k largest logits
if top_k > 0:
cutoff = sorted((v.data for v in nxt), reverse=True)[:top_k][-1]
nxt = [v if v.data >= cutoff else Value(-1e9) for v in nxt]
# 4. softmax -> probabilities
probs = softmax(nxt)
# 5. sample one token from the distribution
r, cumsum = random.random(), 0.0
choice = len(probs) - 1
for i, p in enumerate(probs):
cumsum += p.data
if cumsum >= r:
choice = i
break
tokens.append(choice)
return " ".join(itos.get(t, "<unk>") for t in tokens)
if __name__ == "__main__":
random.seed(0)
model_path = Path("model.bin")
if not model_path.exists():
raise SystemExit("model.bin not found โ run train.py first")
params, vocab, step, loss = load_model(model_path)
print(f"Loaded: step={step} loss={loss:.3f}")
if len(sys.argv) > 1:
print(generate(params, vocab, sys.argv[1]))
else:
temperature, top_k = 0.8, 40
print("Type a prompt. :t 0.9 sets temperature, :k 20 sets top-k, empty line quits.")
while True:
try:
line = input("> ")
except (EOFError, KeyboardInterrupt):
break
if line.strip() == "" or line.strip() in ("exit", "quit"):
break
if line.startswith(":t "):
temperature = float(line[3:]); continue
if line.startswith(":k "):
top_k = int(line[3:]); continue
print(generate(params, vocab, line, temperature=temperature, top_k=top_k))
print()
build_corpus.py โ unchanged from the previous lesson
#!/usr/bin/env python3
"""build_corpus.py โ concatenate .md/.txt files into corpus.txt."""
import argparse, re
from pathlib import Path
def build_corpus(src, out):
files = sorted(src.rglob("*")) if src.is_dir() else [src]
parts = []
for f in files:
if not f.is_file() or f.suffix.lower() not in {".md", ".txt"}:
continue
text = f.read_text(encoding="utf-8")
# normalize whitespace so tokenization is stable
text = re.sub(r"[ \t]+", " ", text)
text = re.sub(r"\n{3,}", "\n\n", text).strip()
if text:
parts.append(text)
if not parts:
raise SystemExit(f"No .md/.txt files found under {src}")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text("\n\n".join(parts), encoding="utf-8")
total = sum(len(p) for p in parts)
print(f"Corpus: {out} ({len(parts)} files, {total:,} chars)")
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("src", type=Path)
ap.add_argument("out", type=Path)
args = ap.parse_args()
build_corpus(args.src, args.out)
Want to smoke-test first โ set STEPS = 5 (about 30 seconds in my container).
Loss stays near 7.6 (ln of the vocab size) โ the model isn't learning; check that LR and the corpus are what you think they are.
Output is repetitive โ raise temperature in the REPL: :t 0.9 or :t 1.0.
Output looks like word salad โ expected at this size and training length. More steps, a bigger model, or a longer context all help, and all cost time.
Out of memory (peak is ~1.5 GB) โ lower BATCH_SIZE to 2 or CTX_LEN; memory grows with batch ร context ร vocab.
Most words are <unk> โ increase VOCAB_SIZE (at the cost of a bigger embedding table).
What you just learned
1. Tokenization is a design choice, not a given. In the previous character-level lesson it was invisible; here it's the biggest decision. Vocabulary size, context length, and how to handle unknowns all flow from it.
2. The architecture doesn't care. Same block, same optimizer, and an autograd engine that needed a single fix (iterative backward(), because a 2,001-way softmax makes the graph deeper than Python's recursion limit). Otherwise only the data pipeline moved.
3. Every tokenization choice trades something. Words give readable output but cost more per step and lose unknown-word handling. Characters preserve everything but waste capacity on spelling. Subwords (the next step) try to get both.
You now have two working language models โ one that predicts characters (from the previous lesson), one that predicts words (from this one) โ built from the same ~500 lines of pure Python. The difference between them is a tokenizer, two constants, and one autograd fix. That's the whole lesson: in language modeling, the tokenizer is the first and most consequential decision, and everything downstream is shaped by it.