Training lesson

Pretrain on real text

The subword model learned the alphabet of tokenization, but it trained for 300 steps on 95,000 tokens โ€” enough to prove the code works, not enough to tell you whether it's learning. This lesson adds the four pieces of training infrastructure that every real run uses โ€” a train/validation split, gradient accumulation, a learning-rate schedule, and checkpointing โ€” at a scale you can read and run. The model is still a toy. The training loop is not.

What stays

The model is a toy

~170K params, 2 layers, 32-dim. Transformer blocks, attention, LayerNorm, GELU, autograd, BPE tokenizer โ€” all unchanged. This lesson does not make the model bigger.

What changes

The loop is real

Val loss every 100 steps, gradient accumulation over 8 microbatches, cosine LR schedule with warmup, checkpoints every 500 steps. These are the same techniques GPT-class runs use โ€” at toy scale.

What it costs

More time, more data

A ~1M-token corpus and 3,000 steps instead of 95K tokens and 300 steps. ~2โ€“3 hours on one core, resumable after a crash.

val loss = the honest number gradient accumulation ยท warmup + cosine decay ยท checkpoints you can resume from
Begin โ†“

Why real data โ€” the toy run's blind spots

The subword lesson trained on a 500K-character corpus and 95,000 tokens. That's enough to prove the tokenizer round-trips text and the model learns something. It is not enough to tell you whether the model is actually learning, or just memorizing the corpus.

A toy training run has three blind spots that a real run fixes:

  • No validation. You measured training loss, which always goes down โ€” even when the model is memorizing noise. Without a held-out set, you can't tell learning from memorization.
  • Tiny effective batch. BATCH_SIZE = 4 sequences of 32 tokens is a batch of 128 tokens. Gradient estimates are noisy, and the model plateaus early. Real runs use batches 100โ€“1,000ร— larger.
  • Constant learning rate. A fixed LR = 3e-3 is too high at the end (the model bounces around the minimum) and arguably too low at the start (it could take bigger steps while the loss is far from optimal). Schedules fix both.

This lesson fixes all three. The model architecture is unchanged. The tokenizer is unchanged. What changes is how you run the training.

Subword lesson (toy)This lesson (real loop)
Corpus500K chars / 95K tokens~5M chars / ~1M tokens
Train/val splitnone90% / 10%
Batch size4 sequences4 ร— 8 accumulation = 32
Steps3003,000
LR scheduleconstant 3e-3warmup 100 โ†’ cosine decay
Val lossnever measuredevery 100 steps
Checkpointsfinal onlyevery 500 steps + final
Runtime~25 min~2โ€“3 h (resumable)
Where to get a real corpus

TinyStories (~2M short children's stories, ~500MB) is the best choice for a model this small โ€” simple vocabulary, coherent grammar, short sentences. Download a 5โ€“10MB slice with curl or from HuggingFace. Other good options: a Project Gutenberg text dump (classic novels), a WikiText-2 slice, or a few hundred .md files from any open-source repo. The corpus loader is the same for all of them โ€” it reads every .txt/.md file under a directory and concatenates them.

Why not just use the tiny corpus? Because the point of this lesson is to see the training curve on data the model can't memorize. On a 500K-char corpus, 3,000 steps will overfit hard and the val loss will rise โ€” which is also instructive, but it's not the main event.

Train/validation split โ€” the honest number

The first piece of infrastructure is the simplest and the most important: hold out 10% of the corpus and never train on it. Every 100 steps, measure the loss on the held-out set. That number โ€” validation loss โ€” is the only number that tells you whether the model is learning or memorizing.

train loss = loss on data the model sees (always goes down)
val loss = loss on data the model never sees (goes down, then up when overfitting)
gap = val loss โˆ’ train loss (grows when overfitting)

The split is done at the token level, not the file level. Concatenate the corpus, tokenize it once, then split the token list 90/10. The val set is the last 10% of tokens โ€” the same convention used in real pretraining.

data.py โ€” the new module
"""
data.py โ€” corpus loading, tokenization, and the train/val split.

Concatenates every text file under `input_dir`, tokenizes once with the
BPE tokenizer, and splits the token stream 90/10. The val set is the
last 10% of tokens, so the model never sees it during training.
"""
from pathlib import Path
from tokenizer import train_bpe, BPETokenizer

TEXT_EXTS = {".txt", ".md", ".rst"}
VAL_FRACTION = 0.1
N_MERGES = 2000


def load_corpus(input_dir):
    """Read every text file under input_dir, concatenate with blank lines."""
    src = Path(input_dir)
    if not src.exists():
        raise SystemExit(f"input dir not found: {src}")
    files = sorted(p for p in src.rglob("*")
                   if p.is_file() and p.suffix.lower() in TEXT_EXTS)
    if not files:
        raise SystemExit(f"no text files under {src}")
    parts = []
    for p in files:
        try:
            t = p.read_text(encoding="utf-8")
        except UnicodeDecodeError:
            t = p.read_text(encoding="latin-1")
        t = t.strip()
        if t:
            parts.append(t)
    return "\n\n".join(parts)


def build_dataset(input_dir, n_merges=N_MERGES, val_fraction=VAL_FRACTION):
    """Returns (train_tokens, val_tokens, tokenizer)."""
    text = load_corpus(input_dir)
    print(f"Corpus: {len(text):,} chars")

    print(f"Training BPE ({n_merges} merges)...")
    merges = train_bpe(text, n_merges)
    tok = BPETokenizer(merges)
    print(f"Vocabulary: {tok.vocab_size} tokens")

    all_ids = tok.encode(text)
    n_val = int(len(all_ids) * val_fraction)
    n_train = len(all_ids) - n_val
    train_ids = all_ids[:n_train]
    val_ids = all_ids[n_train:]
    print(f"Tokens: {len(all_ids):,} total  ({n_train:,} train / {n_val:,} val)")
    return train_ids, val_ids, tok

load_corpus walks the input directory, reads every .txt/.md/.rst file, and joins them with blank lines. It falls back to latin-1 on decode errors so a single bad file doesn't kill the run.

build_dataset does the tokenization once and splits the token list. The val set is the last 10% โ€” this is the standard convention because the corpus is read in file order, so the val set is a contiguous chunk the model never trained on. Splitting randomly would leak information across the boundary.

Why split after tokenization, not before? Because BPE merges are learned from the whole corpus. If you split raw text first and trained BPE only on the train half, you'd get a different tokenizer โ€” and real pretraining always learns the tokenizer on the full corpus (or on a large sample), then splits tokens. The tiny amount of leakage this causes is negligible and is what every real pipeline does.

The val set must be large enough to be meaningful

With 1M tokens total, the val set is ~100K tokens โ€” enough for a stable loss estimate (a few hundred batches of 32ร—32 tokens each). If you use a smaller corpus, either lower VAL_FRACTION to 0.05 or accept that the val loss will be noisy. A val loss computed on 1,000 tokens is not a reliable signal.

Gradient accumulation โ€” bigger batches without more memory

The model is trained with BATCH_SIZE = 4 sequences per step. That's a batch of 128 tokens. The gradient is noisy โ€” each step the model takes a slightly random direction, and the loss curve is jagged. The fix is a bigger effective batch, but you can't just raise BATCH_SIZE: memory grows linearly with batch size, and the backward pass is the expensive part.

Gradient accumulation solves this. Instead of updating the weights after every microbatch, accumulate gradients over ACCUM_STEPS microbatches, then update once. The math is identical to a batch of BATCH_SIZE ร— ACCUM_STEPS โ€” you just paid for it in time, not memory.

for each micro in 0..ACCUM_STEPSโˆ’1:
  loss += compute_loss(forward(x_micro), y_micro) / ACCUM_STEPS
  loss.backward() # grads accumulate into .grad
optimizer.step() # one update per ACCUM_STEPS microbatches
zero_grad() # reset for the next accumulation window

The key detail: loss.backward() adds to .grad (it doesn't overwrite). So calling it ACCUM_STEPS times before optimizer.step() gives you the sum of the microbatches' gradients โ€” exactly the gradient of the concatenated batch. Dividing by ACCUM_STEPS keeps the loss scale comparable to a single-batch step, which matters for the LR schedule.

train.py โ€” the accumulation loop
BATCH_SIZE  = 4    # sequences per microbatch
ACCUM_STEPS = 8    # microbatches per optimizer step โ†’ effective batch 32

for step in range(1, STEPS + 1):
    step_loss = Value(0.0)
    for _ in range(ACCUM_STEPS):
        x, y = sample_batch(train_ids)
        micro = Value(0.0)
        for xi, yi in zip(x, y):
            micro = micro + compute_loss(forward(xi, params), yi)
        micro = micro / BATCH_SIZE
        step_loss = step_loss + micro / ACCUM_STEPS
        micro.backward()   # grads accumulate in .grad

    # one optimizer step per ACCUM_STEPS microbatches
    for i, p in enumerate(plist):
        g = p.grad
        m[i] = b1 * m[i] + (1 - b1) * g
        v[i] = b2 * v[i] + (1 - b2) * g * g
        m_hat = m[i] / (1 - b1 ** step)
        v_hat = v[i] / (1 - b2 ** step)
        p.data -= lr * m_hat / (v_hat ** 0.5 + eps)
        p.grad  = 0.0

Why micro.backward() inside the loop, not step_loss.backward() at the end? Because step_loss is a sum of ACCUM_STEPS separate forward graphs. Calling backward() on the sum would require all of them to still be alive โ€” which means holding ACCUM_STEPS forward graphs in memory at once, defeating the purpose. Calling backward() per microbatch frees each graph immediately, and the gradients accumulate in .grad. The final gradient is the same either way.

Why divide by ACCUM_STEPS? So step_loss is the mean loss over the effective batch, not the sum. This keeps the loss scale independent of ACCUM_STEPS, so the LR schedule doesn't need to change when you change accumulation. It's also what makes the logged loss comparable across runs.

The memory win: peak memory is now one microbatch's forward+backward graph, not ACCUM_STEPS of them. On this model the difference is small (~1.6 GB โ†’ ~1.8 GB, dominated by the bigger corpus), but on a real model accumulation is the difference between "fits on one GPU" and "doesn't."

How to choose ACCUM_STEPS

Target an effective batch of 32โ€“128 sequences for a model this size. With BATCH_SIZE=4, that means ACCUM_STEPS between 8 and 32. Higher is smoother but slower per optimizer step. Start at 8 (effective batch 32) and raise it if the loss curve is noisy. The wall-clock cost is linear in ACCUM_STEPS โ€” you're trading time for gradient quality.

The learning-rate schedule โ€” warmup + cosine decay

A constant learning rate is wrong at both ends of training. At the start, when the loss is far from optimal, you could take bigger steps. At the end, when the model is near a minimum, a constant LR bounces it around and prevents fine-tuning. The standard fix is a schedule: ramp up over the first few hundred steps, then decay smoothly to near zero.

warmup (step < WARMUP):
  lr = LR_MAX ร— step / WARMUP

cosine decay (step โ‰ฅ WARMUP):
  progress = (step โˆ’ WARMUP) / (STEPS โˆ’ WARMUP)
  lr = LR_MIN + 0.5 ร— (LR_MAX โˆ’ LR_MIN) ร— (1 + cos(ฯ€ ร— progress))

Try it: shape the schedule

Drag the warmup slider and watch the learning rate curve. The cosine decay is fixed to the total step count; warmup just delays the start of the decay.

train.py โ€” the schedule
import math

LR_MAX    = 3e-3
LR_MIN    = 3e-4    # 10% of max โ€” never decays to exactly zero
WARMUP    = 100
STEPS     = 3000

def lr_at(step):
    """Cosine schedule with linear warmup. step is 1-indexed."""
    if step <= WARMUP:
        return LR_MAX * step / WARMUP
    progress = (step - WARMUP) / max(1, STEPS - WARMUP)
    return LR_MIN + 0.5 * (LR_MAX - LR_MIN) * (1 + math.cos(math.pi * progress))

Warmup exists because at step 1, the weights are random and the gradients are large and noisy. A full-size LR at step 1 can blow the model up (loss โ†’ NaN). Ramping from 0 to LR_MAX over 100 steps gives the model time to settle into a sensible region before taking big steps. This is standard in every real training run.

Cosine decay is the most common choice because it's smooth at both ends: the derivative is zero at the top (so the transition out of warmup is gentle) and zero at the bottom (so the final steps are tiny and the model can fine-tune). Linear decay works too but is slightly worse in practice; step decay (divide by 10 at fixed steps) is the old-school approach and is fine for small runs.

LR_MIN = 3e-4, not 0. A small residual LR keeps the model learning slowly at the end, which helps on a small corpus where the loss is still improving. Setting LR_MIN = 0 means the last 10% of training does nothing.

What the schedule buys you

Without a schedule, the loss curve on this model typically plateaus around step 1,500โ€“2,000 and then oscillates โ€” the constant LR is too big to settle. With warmup + cosine, the loss keeps improving until the last few hundred steps, and the final loss is typically 0.1โ€“0.2 lower than the constant-LR run at the same step count. That's a small but reliable win, and it's free โ€” the schedule is five lines of code.

Checkpointing โ€” survive a crash, resume where you left off

A 3,000-step run takes 2โ€“3 hours on one core. If your machine reboots, or you hit Ctrl-C, or Python segfaults at step 2,847, you lose everything. Checkpointing fixes this: every 500 steps, write the model weights, the optimizer state, and the current step number to disk. On startup, if a checkpoint exists, load it and resume.

The checkpoint needs four things, not just the weights:

  1. The model weights โ€” the same float32 array the existing save_model writes.
  2. The optimizer state โ€” Adam's m and v buffers. Without these, resuming is not a true continuation: the optimizer restarts from zero momentum and the loss spikes.
  3. The step number โ€” so the LR schedule and the log pick up where they left off.
  4. The tokenizer merges โ€” already saved as model.tok.json, but the checkpoint should write it too so a resumed run uses the same tokenizer.
train.py โ€” checkpoint save/load
import struct
from pathlib import Path

def save_checkpoint(path, params, m, v, step, loss, tok):
    """Atomic write: tmp file + rename, so a crash mid-write doesn't corrupt."""
    tmp = path.with_suffix(path.suffix + ".tmp")
    with tmp.open("wb") as f:
        f.write(struct.pack("<IIIf", step, len(m), len(v), loss))
        for p in all_params(params):
            f.write(struct.pack("<f", p.data))
        for x in m:
            f.write(struct.pack("<f", x))
        for x in v:
            f.write(struct.pack("<f", x))
    tmp.replace(path)
    tok.save(path.with_suffix(".tok.json"))


def load_checkpoint(path, params, tok):
    """Returns (step, m, v, loss) or None if no checkpoint exists."""
    if not path.exists():
        return None
    with path.open("rb") as f:
        step, n_m, n_v, loss = struct.unpack("<IIIf", f.read(16))
        for p in all_params(params):
            p.data = struct.unpack("<f", f.read(4))[0]
        m = [struct.unpack("<f", f.read(4))[0] for _ in range(n_m)]
        v = [struct.unpack("<f", f.read(4))[0] for _ in range(n_v)]
    return step, m, v, loss

Atomic write via tmp.replace(path) โ€” if the process dies mid-write, the original checkpoint is untouched. Writing directly to path would corrupt it and you'd lose the last good checkpoint too. This is a one-line habit that saves runs.

Why save m and v? Adam's update is m_hat / sqrt(v_hat), and m/v are running averages. Resuming without them means the first few hundred steps after resume use wrong per-parameter scales โ€” the loss visibly spikes and takes a while to recover. Real training frameworks always checkpoint optimizer state.

Why save the step? The LR schedule is a function of step. Resuming at step 2,500 with a fresh step=1 would restart the warmup and re-ramp the LR โ€” catastrophic. The step is the schedule's only input.

What about the tokenizer? The checkpoint writes model.tok.json next to it. On resume, load_checkpoint uses the same tok object the run started with โ€” but the file is there so a fresh process can rebuild the tokenizer before calling load_checkpoint. The full train.py listing below shows the order.

The resume test

Run python train.py corpus/, let it reach step ~600, Ctrl-C. Run it again. It should print Resuming from step 500 (loss 5.42) and continue from there โ€” not from step 1. The loss at step 501 should be close to the loss at step 500, not spiking. If it spikes, your optimizer state wasn't saved correctly.

Reading the curve โ€” what healthy training looks like

With validation loss measured every 100 steps, the training log becomes a diagnostic tool. Here's what the four phases of a healthy run look like:

Loss curve โ€” train vs. validation (illustrative)
8.0 7.0 6.0 5.0 4.0 0 1000 2000 3000 train val
Four phases: (1) warmup โ€” both losses drop fast; (2) main training โ€” steady decline; (3) plateau โ€” the model has learned the easy structure; (4) fine-tuning โ€” the cosine schedule lets it settle. If the val curve turns up while train keeps going down, you're overfitting.

The four phases

  1. Warmup (steps 0โ€“100). Both losses drop steeply. The LR is ramping up, so this is fast. If you see a spike here, the warmup is too short or LR_MAX is too high.
  2. Main training (steps 100โ€“1,500). The steep drop flattens into a steady decline. Train and val losses track each other closely โ€” this is the model learning generalizable structure. If val diverges here, the model is overfitting early; add data or reduce capacity.
  3. Plateau (steps 1,500โ€“2,500). The curve flattens. The model has learned the easy patterns and is now working on the hard ones. This is where most of the wall-clock time goes, and it's normal.
  4. Fine-tuning (steps 2,500โ€“3,000). The cosine schedule reduces the LR to near zero, and the loss drops another small amount as the model settles into a minimum. If the val loss turns up here while train keeps going down, you're overfitting โ€” stop earlier.
SymptomLikely causeFix
Loss spikes at step 1โ€“20LR too high / warmup too shortIncrease WARMUP, lower LR_MAX
Loss โ†’ NaNGradient blowupLower LR_MAX; add grad clipping (optional)
Val loss rises while train fallsOverfittingMore data, lower capacity, or early stop
Loss plateaus immediatelyLR too low / bad initRaise LR_MAX; check init_model scale
Loss jitters but doesn't fallBatch too small / accumulation too lowRaise ACCUM_STEPS
Resume causes a loss spikeOptimizer state not savedCheck save_checkpoint includes m/v
The experiment that makes this concrete

Run the same corpus twice: once with the schedule and accumulation from this lesson, once with the constant LR and BATCH_SIZE=4 from the subword lesson. Plot both val-loss curves on the same axes. The scheduled run should reach a lower final val loss and have a smoother curve. This is the single most useful comparison in the whole series โ€” it shows that training methodology matters as much as architecture at this scale.

Where this leads

This lesson gave you the training infrastructure that every real run uses. The model is still ~170K parameters โ€” a toy โ€” but it's now trained honestly: with a validation signal, a proper schedule, and the ability to resume. From here, the path to a model that writes coherent text is scale, not new training concepts:

StageModelDataTraining
Subword (lesson 3)~170K params, 2 layers95K tokens300 steps, constant LR
Pretrained (this lesson)~170K params, 2 layers~1M tokens3,000 steps, schedule + accumulation
Small GPT-2 clone~10M params, 6 layers~100M tokens50K steps โ€” needs PyTorch
GPT-2 small124M params, 12 layers~10B tokens300K steps โ€” GPUs

The next lesson is sampling and decoding strategies โ€” top-p, repetition penalty, temperature annealing โ€” using the checkpoint you just trained. That lesson is small, self-contained, and immediately visible in the output. After that, the honest next step is a PyTorch port: the pure-Python engine can't reasonably train a 10M-param model, but the concepts (schedule, accumulation, checkpointing, validation) transfer directly.

The source files

To run it

Requires: Python 3.9+. No pip install needed. No dependencies beyond the standard library.

Layout: seven files in one folder, plus a corpus/ directory of text files. Every file is listed in full below. train.py writes model.bin, model.tok.json, and model.ckpt.bin; infer.py loads model.bin and model.tok.json.

# 1. get a corpus (TinyStories slice, Gutenberg, or any .md/.txt files)
mkdir corpus && cp your_texts/*.txt corpus/
python train.py corpus/
python infer.py

Expected: ~170K parameters, ~1M tokens (depends on corpus), 3,000 steps. BPE training takes ~2โ€“3 minutes on 5M chars; training takes ~2โ€“3 hours on one core, resumable via model.ckpt.bin. Peak memory ~1.8 GB.

tokenizer.py โ€” complete (unchanged from lesson 3)

tokenizer.py
"""
tokenizer.py - byte-level BPE, learned from a corpus.

Token ids 0-255 are raw bytes. Merge number k creates id 256 + k.
Text is first split into chunks (a word with its leading space, a number,
a punctuation run, or whitespace) and merges never cross chunk borders.
Because whitespace is kept, decode(encode(text)) == text exactly.

No external dependencies. Pure Python.
"""
import json
import re
from collections import Counter

PRETOKEN = re.compile(r" ?[^\W\d]+| ?\d+| ?[^\s\w]+|\s+(?!\S)|\s+")


def pretokenize(text):
    return PRETOKEN.findall(text)


def merge_ids(ids, pair, new_id):
    """Replace every adjacent occurrence of `pair` in `ids` with `new_id`."""
    out, i = [], 0
    while i < len(ids):
        if i < len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]:
            out.append(new_id)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out


def train_bpe(text, n_merges=2000):
    """Learn a merge table from raw text. Returns a list of (id, id) pairs."""
    chunks = pretokenize(text)
    if not chunks:
        raise ValueError("empty corpus")

    freq = Counter(tuple(c.encode("utf-8")) for c in chunks)

    merges = []
    for _ in range(n_merges):
        pairs = Counter()
        for word, count in freq.items():
            for pair in zip(word, word[1:]):
                pairs[pair] += count
        if not pairs:
            break
        best = max(pairs, key=pairs.get)
        if pairs[best] < 2:
            break
        new_id = 256 + len(merges)
        merges.append(best)

        new_freq = Counter()
        for word, count in freq.items():
            new_freq[tuple(merge_ids(word, best, new_id))] += count
        freq = new_freq

    return merges


class BPETokenizer:
    """A byte-level BPE tokenizer."""

    def __init__(self, merges):
        self.merges = [tuple(p) for p in merges]
        self.ranks = {p: i for i, p in enumerate(self.merges)}
        self.itos = {i: bytes([i]) for i in range(256)}
        for i, (a, b) in enumerate(self.merges):
            self.itos[256 + i] = self.itos[a] + self.itos[b]
        self.vocab_size = len(self.itos)
        self._cache = {}

    def _encode_chunk(self, chunk):
        if chunk in self._cache:
            return self._cache[chunk]
        ids = list(chunk.encode("utf-8"))
        while len(ids) > 1:
            best = min(zip(ids, ids[1:]), key=lambda p: self.ranks.get(p, float("inf")))
            if best not in self.ranks:
                break
            ids = merge_ids(ids, best, 256 + self.ranks[best])
        self._cache[chunk] = ids
        return ids

    def encode(self, text):
        ids = []
        for chunk in pretokenize(text):
            ids.extend(self._encode_chunk(chunk))
        return ids

    def decode(self, ids):
        raw = b"".join(self.itos[i] for i in ids)
        return raw.decode("utf-8", errors="replace")

    def save(self, path):
        with open(path, "w", encoding="utf-8") as f:
            json.dump(self.merges, f)

    @classmethod
    def load(cls, path):
        with open(path, encoding="utf-8") as f:
            return cls(json.load(f))

micrograd.py โ€” complete (unchanged)

micrograd.py
"""
micrograd.py โ€” a tiny autograd engine.
backward() is iterative, which matters because a 2,256-way softmax
makes the graph deep enough to hit Python's recursion limit.
"""
import math


class Value:
    def __init__(self, data, children=(), op=""):
        self.data      = data
        self.grad      = 0.0
        self._prev     = children
        self._op       = op
        self._backward = lambda: None

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad  += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad  += other.data * out.grad
            other.grad += self.data  * out.grad
        out._backward = _backward
        return out

    def __pow__(self, n):
        out = Value(self.data ** n, (self,), f"**{n}")
        def _backward():
            self.grad += n * (self.data ** (n - 1)) * out.grad
        out._backward = _backward
        return out

    def exp(self):
        out = Value(math.exp(self.data), (self,), "exp")
        def _backward():
            self.grad += out.data * out.grad
        out._backward = _backward
        return out

    def log(self):
        x = max(self.data, 1e-30)
        out = Value(math.log(x), (self,), "log")
        def _backward():
            self.grad += (1.0 / x) * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        out = Value(math.tanh(self.data), (self,), "tanh")
        def _backward():
            self.grad += (1.0 - out.data ** 2) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo, visited = [], set()
        stack = [(self, False)]
        while stack:
            v, expanded = stack.pop()
            if expanded:
                topo.append(v)
                continue
            if v in visited:
                continue
            visited.add(v)
            stack.append((v, True))
            for child in v._prev:
                if child not in visited:
                    stack.append((child, False))
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()
            v._backward = lambda: None
            v._prev = ()
        topo.clear(); visited.clear()


def __radd__(self, other): return self + other
def __rmul__(self, other): return self * other
def __neg__(self): return self * -1
def __sub__(self, other): return self + (-other)
def __truediv__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    return self * (other ** -1)
def __rsub__(self, other): return (-self) + other
def __rtruediv__(self, other): return Value(other) * (self ** -1)
Value.__rsub__    = __rsub__
Value.__rtruediv__ = __rtruediv__
Value.__radd__    = __radd__
Value.__rmul__    = __rmul__
Value.__neg__     = __neg__
Value.__sub__     = __sub__
Value.__truediv__ = __truediv__


def dot(xs, ws):
    out = Value(sum(x.data * w.data for x, w in zip(xs, ws)),
                tuple(xs) + tuple(ws), "dot")
    def _backward():
        g = out.grad
        for x, w in zip(xs, ws):
            x.grad += w.data * g
            w.grad += x.data * g
    out._backward = _backward
    return out

model.py โ€” complete (unchanged from lesson 3)

model.py
"""
model.py โ€” the tiny Transformer. Identical to the subword lesson.
CTX_LEN = 32 (subword tokens), vocab = 256 + n_merges.
all_params() flattens every matrix into individual Value objects.
"""
import math, random, struct
from micrograd import Value, dot

CTX_LEN  = 32
D_MODEL  = 32
N_LAYERS = 2
N_HEADS  = 2
D_FF     = 128
HEAD_DIM = D_MODEL // N_HEADS


def init_matrix(rows, cols, scale=0.08):
    return [[Value(random.gauss(0, scale)) for _ in range(cols)] for _ in range(rows)]

def init_vector(n, scale=0.08):
    return [Value(random.gauss(0, scale)) for _ in range(n)]

def init_model(vocab_size):
    p = {}
    p["tok_emb"]  = init_matrix(vocab_size, D_MODEL)
    p["pos_emb"]  = init_matrix(CTX_LEN, D_MODEL)
    p["blocks"]   = []
    for _ in range(N_LAYERS):
        b = {}
        b["ln1_g"] = init_vector(D_MODEL, 0.0)
        b["ln1_b"] = init_vector(D_MODEL, 0.0)
        b["wq"] = init_matrix(D_MODEL, D_MODEL)
        b["wk"] = init_matrix(D_MODEL, D_MODEL)
        b["wv"] = init_matrix(D_MODEL, D_MODEL)
        b["wo"] = init_matrix(D_MODEL, D_MODEL)
        b["ln2_g"] = init_vector(D_MODEL, 0.0)
        b["ln2_b"] = init_vector(D_MODEL, 0.0)
        b["w1"] = init_matrix(D_FF, D_MODEL)
        b["b1"] = init_vector(D_FF, 0.0)
        b["w2"] = init_matrix(D_MODEL, D_FF)
        b["b2"] = init_vector(D_MODEL, 0.0)
        p["blocks"].append(b)
    p["lnf_g"] = init_vector(D_MODEL, 0.0)
    p["lnf_b"] = init_vector(D_MODEL, 0.0)
    p["head"]  = init_matrix(vocab_size, D_MODEL)
    return p


def _flatten(obj):
    if isinstance(obj, Value):
        yield obj
    else:
        for x in obj:
            yield from _flatten(x)

def all_params(p):
    for key in ("tok_emb", "pos_emb"):
        yield from _flatten(p[key])
    for b in p["blocks"]:
        for key in ("ln1_g","ln1_b","wq","wk","wv","wo","ln2_g","ln2_b","w1","b1","w2","b2"):
            yield from _flatten(b[key])
    yield from _flatten(p["lnf_g"])
    yield from _flatten(p["lnf_b"])
    yield from _flatten(p["head"])


def matmul(x, W):
    return [dot(x, row) for row in W]

def add_rows(a, b):
    return [x + y for x, y in zip(a, b)]

def softmax(logits):
    m = max(v.data for v in logits)
    e = [(v - m).exp() for v in logits]
    s = sum(v.data for v in e)
    return [v / s for v in e]

def layernorm(x, g, b, eps=1e-5):
    n = len(x)
    mean = sum(v.data for v in x) / n
    var  = sum((v.data - mean) ** 2 for v in x) / n
    inv  = 1.0 / math.sqrt(var + eps)
    out = []
    for i, v in enumerate(x):
        out.append((v - mean) * inv * g[i] + b[i])
    return out

def gelu_approx(x):
    return 0.5 * x * (1.0 + (math.sqrt(2.0 / math.pi) * (x + 0.044715 * (x ** 3))).tanh())


def attention(x, b):
    Q = [matmul(t, b["wq"]) for t in x]
    K = [matmul(t, b["wk"]) for t in x]
    V = [matmul(t, b["wv"]) for t in x]
    out = []
    for i in range(CTX_LEN):
        heads_out = []
        for h in range(N_HEADS):
            off = h * HEAD_DIM
            scores = []
            for j in range(i + 1):
                s = sum(Q[i][off + d] * K[j][off + d] for d in range(HEAD_DIM))
                scores.append(s / math.sqrt(HEAD_DIM))
            probs = softmax(scores)
            head = [Value(0.0) for _ in range(HEAD_DIM)]
            for j in range(i + 1):
                for d in range(HEAD_DIM):
                    head[d] = head[d] + probs[j] * V[j][off + d]
            heads_out.extend(head)
        out.append(matmul(heads_out, b["wo"]))
    return out

def block(x, b):
    a = [layernorm(t, b["ln1_g"], b["ln1_b"]) for t in x]
    a = attention(a, b)
    x = [add_rows(xi, ai) for xi, ai in zip(x, a)]
    a = [layernorm(t, b["ln2_g"], b["ln2_b"]) for t in x]
    ff = []
    for t in a:
        h = matmul(t, b["w1"])
        h = [h[k] + b["b1"][k] for k in range(D_FF)]
        h = [gelu_approx(v) for v in h]
        o = matmul(h, b["w2"])
        o = [o[k] + b["b2"][k] for k in range(D_MODEL)]
        ff.append(o)
    return [add_rows(xi, fi) for xi, fi in zip(x, ff)]


def forward(tokens, p):
    x = [add_rows(p["tok_emb"][t], p["pos_emb"][i]) for i, t in enumerate(tokens)]
    for b in p["blocks"]:
        x = block(x, b)
    x = [layernorm(t, p["lnf_g"], p["lnf_b"]) for t in x]
    logits = [matmul(t, p["head"]) for t in x]
    return logits


def save_model(path, p, itos, steps, loss):
    with open(path, "wb") as f:
        f.write(struct.pack("<III", steps, len(itos), len(list(all_params(p)))))
        f.write(struct.pack("<f", loss))
        for v in all_params(p):
            f.write(struct.pack("<f", v.data))

def load_model(path):
    with open(path, "rb") as f:
        steps, vocab_size, n_params = struct.unpack("<III", f.read(12))
        loss, = struct.unpack("<f", f.read(4))
        params = []
        for _ in range(n_params):
            params.append(Value(struct.unpack("<f", f.read(4))[0]))
    return steps, vocab_size, loss, params

data.py โ€” complete

data.py
"""
data.py โ€” corpus loading, tokenization, and the train/val split.
"""
from pathlib import Path
from tokenizer import train_bpe, BPETokenizer

TEXT_EXTS = {".txt", ".md", ".rst"}
VAL_FRACTION = 0.1
N_MERGES = 2000


def load_corpus(input_dir):
    """Read every text file under input_dir, concatenate with blank lines."""
    src = Path(input_dir)
    if not src.exists():
        raise SystemExit(f"input dir not found: {src}")
    files = sorted(p for p in src.rglob("*")
                   if p.is_file() and p.suffix.lower() in TEXT_EXTS)
    if not files:
        raise SystemExit(f"no text files under {src}")
    parts = []
    for p in files:
        try:
            t = p.read_text(encoding="utf-8")
        except UnicodeDecodeError:
            t = p.read_text(encoding="latin-1")
        t = t.strip()
        if t:
            parts.append(t)
    return "\n\n".join(parts)


def build_dataset(input_dir, n_merges=N_MERGES, val_fraction=VAL_FRACTION):
    """Returns (train_tokens, val_tokens, tokenizer)."""
    text = load_corpus(input_dir)
    print(f"Corpus: {len(text):,} chars")
    print(f"Training BPE ({n_merges} merges)...")
    merges = train_bpe(text, n_merges)
    tok = BPETokenizer(merges)
    print(f"Vocabulary: {tok.vocab_size} tokens")
    all_ids = tok.encode(text)
    n_val = int(len(all_ids) * val_fraction)
    n_train = len(all_ids) - n_val
    train_ids = all_ids[:n_train]
    val_ids = all_ids[n_train:]
    print(f"Tokens: {len(all_ids):,} total  ({n_train:,} train / {n_val:,} val)")
    return train_ids, val_ids, tok

train.py โ€” complete

train.py
#!/usr/bin/env python3
"""train.py โ€” pretrain the subword model on real text."""
import sys, time, random, gc, math, struct
from pathlib import Path
from tokenizer import BPETokenizer
from micrograd import Value
from model import CTX_LEN, init_model, all_params, forward, softmax, save_model
from data import build_dataset

BATCH_SIZE       = 4
ACCUM_STEPS      = 8
STEPS            = 3000
LR_MAX           = 3e-3
LR_MIN           = 3e-4
WARMUP           = 100
EVAL_EVERY       = 100
CHECKPOINT_EVERY = 500
EVAL_BATCHES     = 20
N_MERGES         = 2000


def compute_loss(logits, targets):
    total = Value(0.0)
    for t in range(len(logits)):
        probs = softmax(logits[t])
        total = total - probs[targets[t]].log()
    return total / len(logits)


def lr_at(step):
    if step <= WARMUP:
        return LR_MAX * step / WARMUP
    progress = (step - WARMUP) / max(1, STEPS - WARMUP)
    return LR_MIN + 0.5 * (LR_MAX - LR_MIN) * (1 + math.cos(math.pi * progress))


def sample_batch(ids):
    x, y = [], []
    for _ in range(BATCH_SIZE):
        j = random.randint(0, len(ids) - CTX_LEN - 1)
        x.append(ids[j : j + CTX_LEN])
        y.append(ids[j + 1 : j + CTX_LEN + 1])
    return x, y


def evaluate(params, val_ids):
    total = 0.0
    for _ in range(EVAL_BATCHES):
        x, y = sample_batch(val_ids)
        for xi, yi in zip(x, y):
            logits = forward(xi, params)
            total += compute_loss(logits, yi).data
    return total / (EVAL_BATCHES * BATCH_SIZE)


def save_checkpoint(path, params, m, v, step, loss, tok):
    tmp = path.with_suffix(path.suffix + ".tmp")
    with tmp.open("wb") as f:
        f.write(struct.pack("<IIIf", step, len(m), len(v), loss))
        for p in all_params(params):
            f.write(struct.pack("<f", p.data))
        for x in m:
            f.write(struct.pack("<f", x))
        for x in v:
            f.write(struct.pack("<f", x))
    tmp.replace(path)
    tok.save(path.with_suffix(".tok.json"))


def load_checkpoint(path, params, tok):
    if not path.exists():
        return None
    with path.open("rb") as f:
        step, n_m, n_v, loss = struct.unpack("<IIIf", f.read(16))
        for p in all_params(params):
            p.data = struct.unpack("<f", f.read(4))[0]
        m = [struct.unpack("<f", f.read(4))[0] for _ in range(n_m)]
        v = [struct.unpack("<f", f.read(4))[0] for _ in range(n_v)]
    return step, m, v, loss


def main():
    random.seed(0)
    gc.disable()
    input_dir = Path(sys.argv[1] if len(sys.argv) > 1 else "corpus")
    out_path  = Path(sys.argv[2] if len(sys.argv) > 2 else "model.bin")
    ckpt_path = out_path.with_suffix(".ckpt.bin")

    train_ids, val_ids, tok = build_dataset(input_dir, n_merges=N_MERGES)
    if len(train_ids) <= CTX_LEN + 1 or len(val_ids) <= CTX_LEN + 1:
        raise SystemExit("Corpus too short after split")

    vocab_size = tok.vocab_size
    params = init_model(vocab_size)
    plist  = list(all_params(params))
    print(f"Params: {len(plist):,}")

    m = [0.0] * len(plist)
    v = [0.0] * len(plist)
    b1, b2, eps = 0.9, 0.999, 1e-8
    start_step = 0

    ckpt = load_checkpoint(ckpt_path, params, tok)
    if ckpt is not None:
        start_step, m, v, last_loss = ckpt
        print(f"Resuming from step {start_step} (loss {last_loss:.4f})")

    t0 = time.time()
    recent_train = []
    print(f"Training {start_step+1}..{STEPS}  (warmup {WARMUP}, LR_MAX {LR_MAX}, accum {ACCUM_STEPS})")

    for step in range(start_step + 1, STEPS + 1):
        lr = lr_at(step)
        step_loss = Value(0.0)
        for _ in range(ACCUM_STEPS):
            x, y = sample_batch(train_ids)
            micro = Value(0.0)
            for xi, yi in zip(x, y):
                micro = micro + compute_loss(forward(xi, params), yi)
            micro = micro / BATCH_SIZE
            step_loss = step_loss + micro / ACCUM_STEPS
            micro.backward()

        for i, p in enumerate(plist):
            g = p.grad
            m[i] = b1 * m[i] + (1 - b1) * g
            v[i] = b2 * v[i] + (1 - b2) * g * g
            m_hat = m[i] / (1 - b1 ** step)
            v_hat = v[i] / (1 - b2 ** step)
            p.data -= lr * m_hat / (v_hat ** 0.5 + eps)
            p.grad  = 0.0

        recent_train.append(step_loss.data)
        if step % EVAL_EVERY == 0 or step == STEPS:
            train_avg = sum(recent_train[-EVAL_EVERY:]) / len(recent_train[-EVAL_EVERY:])
            val_loss = evaluate(params, val_ids)
            print(f"step {step:5d}  lr {lr:.2e}  train {train_avg:.4f}  val {val_loss:.4f}")

        if step % CHECKPOINT_EVERY == 0 or step == STEPS:
            save_checkpoint(ckpt_path, params, m, v, step, step_loss.data, tok)
            print(f"  checkpoint saved at step {step}")

    final_val = evaluate(params, val_ids)
    save_model(out_path, params, tok.itos, STEPS, final_val)
    tok.save(out_path.with_suffix(".tok.json"))
    print(f"Saved: {out_path}  (final val loss: {final_val:.4f})")
    print(f"Done in {(time.time() - t0)/60:.1f} min")


if __name__ == "__main__":
    main()

infer.py โ€” complete (unchanged from lesson 3)

infer.py
#!/usr/bin/env python3
"""infer.py โ€” sample from the subword model."""
import sys, struct, math, random
from pathlib import Path
from tokenizer import BPETokenizer
from micrograd import Value
from model import CTX_LEN, init_model, all_params, forward, softmax


def load_weights_into(model, flat):
    it = iter(flat)
    for v in all_params(model):
        v.data = next(it)


def load_model_bin(path):
    with open(path, "rb") as f:
        steps, vocab_size, n_params = struct.unpack("<III", f.read(12))
        loss, = struct.unpack("<f", f.read(4))
        flat = [struct.unpack("<f", f.read(4))[0] for _ in range(n_params)]
    model = init_model(vocab_size)
    load_weights_into(model, flat)
    return model, steps, vocab_size, loss


def sample(logits, temperature=1.0, top_k=None):
    if temperature <= 0:
        return max(range(len(logits)), key=lambda i: logits[i].data)
    scaled = [v / temperature for v in logits]
    probs = softmax(scaled)
    if top_k:
        idx = sorted(range(len(probs)), key=lambda i: probs[i].data, reverse=True)[:top_k]
        keep = {i: probs[i].data for i in idx}
        z = sum(keep.values())
        r = random.random() * z
        acc = 0.0
        for i, p in keep.items():
            acc += p
            if r <= acc:
                return i
        return idx[-1]
    r = random.random()
    acc = 0.0
    for i, p in enumerate(probs):
        acc += p.data
        if r <= acc:
            return i
    return len(probs) - 1


def _pad_to_ctx(tokens):
    if len(tokens) >= CTX_LEN:
        return tokens[-CTX_LEN:]
    return [32] * (CTX_LEN - len(tokens)) + tokens


def generate(model, tok, prompt, n_tokens=60, temperature=1.0, top_k=None):
    tokens = tok.encode(prompt)
    if not tokens:
        tokens = [32]
    for _ in range(n_tokens):
        ctx = _pad_to_ctx(tokens)
        logits = forward(ctx, model)
        next_id = sample(logits[-1], temperature, top_k)
        tokens.append(next_id)
    return tok.decode(tokens)


def main():
    random.seed(0)
    model_bin = Path(sys.argv[1] if len(sys.argv) > 1 else "model.bin")
    tok_json  = Path(sys.argv[2] if len(sys.argv) > 2 else "model.tok.json")
    if not model_bin.exists():
        raise SystemExit(f"missing {model_bin}; run train.py first")
    if not tok_json.exists():
        raise SystemExit(f"missing {tok_json}; run train.py first")
    model, steps, vocab_size, loss = load_model_bin(model_bin)
    tok = BPETokenizer.load(tok_json)
    if tok.vocab_size != vocab_size:
        raise SystemExit(f"vocab mismatch: model={vocab_size} tokenizer={tok.vocab_size}")
    print(f"Loaded: {steps} steps, vocab {vocab_size}, val loss {loss:.4f}")
    print("Commands: :t <temp>  :k <top_k>  :q  (or type a prompt)")
    temp, topk = 1.0, None
    while True:
        try:
            line = input("> ").strip()
        except (EOFError, KeyboardInterrupt):
            print()
            break
        if not line:
            continue
        if line == ":q":
            break
        if line.startswith(":t "):
            temp = float(line[3:]); print(f"temperature = {temp}"); continue
        if line.startswith(":k "):
            topk = int(line[3:]); print(f"top_k = {topk}"); continue
        out = generate(model, tok, line, n_tokens=60, temperature=temp, top_k=topk)
        print(out)


if __name__ == "__main__":
    main()
Troubleshooting

Smoke test first โ†’ set STEPS = 20, N_MERGES = 100, and point at a tiny corpus. Everything should finish in under a minute and produce model.bin + model.tok.json + model.ckpt.bin.

"Corpus too short after split" โ†’ your corpus is under ~10K tokens. Add files or lower VAL_FRACTION.

Loss spikes at step 1 โ†’ LR_MAX is too high for your model. Try 1e-3 and increase WARMUP to 200.

Resume causes a loss spike โ†’ optimizer state wasn't loaded. Check that the log says "Resuming from step โ€ฆ".

Val loss goes up โ†’ overfitting. Stop earlier, get more data, or reduce N_LAYERS from 2 to 1 as an experiment.

'list' object has no attribute 'grad' โ†’ all_params is yielding matrix rows. Use the full model.py listing above, which flattens via _flatten.

Training is very slow โ†’ pure Python, ~1โ€“2 s/step with ACCUM_STEPS=8. Lower ACCUM_STEPS to 2โ€“4 to iterate faster; raise it back for the real run.

Checkpoint file is huge โ†’ 3 arrays of ~170K float32 โ‰ˆ 2 MB. That's expected. Delete model.ckpt.bin when a run finishes to save space.

What you just learned

Three takeaways

1. Validation loss is the only honest number. Training loss always goes down; val loss tells you whether the model is learning or memorizing. Hold out 10% of the corpus and measure it every 100 steps. Without this, you're flying blind.

2. Training methodology matters as much as architecture. Gradient accumulation, a warmup + cosine schedule, and checkpointing don't change the model โ€” they change how well it's trained. At this scale, that's worth 0.1โ€“0.2 nats of final loss, which is more than most architecture tweaks.

3. Every real run is resumable. Checkpoints that include the optimizer state and the step number mean a crash costs you minutes, not hours. The atomic-write pattern (tmp + rename) is a one-line habit that prevents the worst failure mode: a corrupted checkpoint when you need it most.

You now have a properly trained subword model โ€” not just a demo that runs, but a run you can trust: it has a validation signal, a schedule that lets it settle, and checkpoints that make it survive. The model is still tiny (~170K parameters, 2 layers), but the training loop is now the same shape as GPT-2's, just smaller. The next lesson โ€” sampling and decoding โ€” uses this checkpoint and shows how the inference side of the loop works: temperature, top-k, top-p, repetition penalty, and why the same model can produce wildly different output depending on how you sample from it.