2
Train/validation split โ the honest number
The first piece of infrastructure is the simplest and the most important: hold out 10% of the corpus and never train on it. Every 100 steps, measure the loss on the held-out set. That number โ validation loss โ is the only number that tells you whether the model is learning or memorizing.
train loss = loss on data the model sees (always goes down)
val loss = loss on data the model never sees (goes down, then up when overfitting)
gap = val loss โ train loss (grows when overfitting)
The split is done at the token level, not the file level. Concatenate the corpus, tokenize it once, then split the token list 90/10. The val set is the last 10% of tokens โ the same convention used in real pretraining.
"""
data.py โ corpus loading, tokenization, and the train/val split.
Concatenates every text file under `input_dir`, tokenizes once with the
BPE tokenizer, and splits the token stream 90/10. The val set is the
last 10% of tokens, so the model never sees it during training.
"""
from pathlib import Path
from tokenizer import train_bpe, BPETokenizer
TEXT_EXTS = {".txt", ".md", ".rst"}
VAL_FRACTION = 0.1
N_MERGES = 2000
def load_corpus(input_dir):
"""Read every text file under input_dir, concatenate with blank lines."""
src = Path(input_dir)
if not src.exists():
raise SystemExit(f"input dir not found: {src}")
files = sorted(p for p in src.rglob("*")
if p.is_file() and p.suffix.lower() in TEXT_EXTS)
if not files:
raise SystemExit(f"no text files under {src}")
parts = []
for p in files:
try:
t = p.read_text(encoding="utf-8")
except UnicodeDecodeError:
t = p.read_text(encoding="latin-1")
t = t.strip()
if t:
parts.append(t)
return "\n\n".join(parts)
def build_dataset(input_dir, n_merges=N_MERGES, val_fraction=VAL_FRACTION):
"""Returns (train_tokens, val_tokens, tokenizer)."""
text = load_corpus(input_dir)
print(f"Corpus: {len(text):,} chars")
print(f"Training BPE ({n_merges} merges)...")
merges = train_bpe(text, n_merges)
tok = BPETokenizer(merges)
print(f"Vocabulary: {tok.vocab_size} tokens")
all_ids = tok.encode(text)
n_val = int(len(all_ids) * val_fraction)
n_train = len(all_ids) - n_val
train_ids = all_ids[:n_train]
val_ids = all_ids[n_train:]
print(f"Tokens: {len(all_ids):,} total ({n_train:,} train / {n_val:,} val)")
return train_ids, val_ids, tok
load_corpus walks the input directory, reads every .txt/.md/.rst file, and joins them with blank lines. It falls back to latin-1 on decode errors so a single bad file doesn't kill the run.
build_dataset does the tokenization once and splits the token list. The val set is the last 10% โ this is the standard convention because the corpus is read in file order, so the val set is a contiguous chunk the model never trained on. Splitting randomly would leak information across the boundary.
Why split after tokenization, not before? Because BPE merges are learned from the whole corpus. If you split raw text first and trained BPE only on the train half, you'd get a different tokenizer โ and real pretraining always learns the tokenizer on the full corpus (or on a large sample), then splits tokens. The tiny amount of leakage this causes is negligible and is what every real pipeline does.
โ ๏ธ
The val set must be large enough to be meaningful
With 1M tokens total, the val set is ~100K tokens โ enough for a stable loss estimate (a few hundred batches of 32ร32 tokens each). If you use a smaller corpus, either lower VAL_FRACTION to 0.05 or accept that the val loss will be noisy. A val loss computed on 1,000 tokens is not a reliable signal.
3
Gradient accumulation โ bigger batches without more memory
The model is trained with BATCH_SIZE = 4 sequences per step. That's a batch of 128 tokens. The gradient is noisy โ each step the model takes a slightly random direction, and the loss curve is jagged. The fix is a bigger effective batch, but you can't just raise BATCH_SIZE: memory grows linearly with batch size, and the backward pass is the expensive part.
Gradient accumulation solves this. Instead of updating the weights after every microbatch, accumulate gradients over ACCUM_STEPS microbatches, then update once. The math is identical to a batch of BATCH_SIZE ร ACCUM_STEPS โ you just paid for it in time, not memory.
for each micro in 0..ACCUM_STEPSโ1:
loss += compute_loss(forward(x_micro), y_micro) / ACCUM_STEPS
loss.backward() # grads accumulate into .grad
optimizer.step() # one update per ACCUM_STEPS microbatches
zero_grad() # reset for the next accumulation window
The key detail: loss.backward() adds to .grad (it doesn't overwrite). So calling it ACCUM_STEPS times before optimizer.step() gives you the sum of the microbatches' gradients โ exactly the gradient of the concatenated batch. Dividing by ACCUM_STEPS keeps the loss scale comparable to a single-batch step, which matters for the LR schedule.
BATCH_SIZE = 4
ACCUM_STEPS = 8
for step in range(1, STEPS + 1):
step_loss = Value(0.0)
for _ in range(ACCUM_STEPS):
x, y = sample_batch(train_ids)
micro = Value(0.0)
for xi, yi in zip(x, y):
micro = micro + compute_loss(forward(xi, params), yi)
micro = micro / BATCH_SIZE
step_loss = step_loss + micro / ACCUM_STEPS
micro.backward()
for i, p in enumerate(plist):
g = p.grad
m[i] = b1 * m[i] + (1 - b1) * g
v[i] = b2 * v[i] + (1 - b2) * g * g
m_hat = m[i] / (1 - b1 ** step)
v_hat = v[i] / (1 - b2 ** step)
p.data -= lr * m_hat / (v_hat ** 0.5 + eps)
p.grad = 0.0
Why micro.backward() inside the loop, not step_loss.backward() at the end? Because step_loss is a sum of ACCUM_STEPS separate forward graphs. Calling backward() on the sum would require all of them to still be alive โ which means holding ACCUM_STEPS forward graphs in memory at once, defeating the purpose. Calling backward() per microbatch frees each graph immediately, and the gradients accumulate in .grad. The final gradient is the same either way.
Why divide by ACCUM_STEPS? So step_loss is the mean loss over the effective batch, not the sum. This keeps the loss scale independent of ACCUM_STEPS, so the LR schedule doesn't need to change when you change accumulation. It's also what makes the logged loss comparable across runs.
The memory win: peak memory is now one microbatch's forward+backward graph, not ACCUM_STEPS of them. On this model the difference is small (~1.6 GB โ ~1.8 GB, dominated by the bigger corpus), but on a real model accumulation is the difference between "fits on one GPU" and "doesn't."
๐งฎ
How to choose ACCUM_STEPS
Target an effective batch of 32โ128 sequences for a model this size. With BATCH_SIZE=4, that means ACCUM_STEPS between 8 and 32. Higher is smoother but slower per optimizer step. Start at 8 (effective batch 32) and raise it if the loss curve is noisy. The wall-clock cost is linear in ACCUM_STEPS โ you're trading time for gradient quality.
5
Checkpointing โ survive a crash, resume where you left off
A 3,000-step run takes 2โ3 hours on one core. If your machine reboots, or you hit Ctrl-C, or Python segfaults at step 2,847, you lose everything. Checkpointing fixes this: every 500 steps, write the model weights, the optimizer state, and the current step number to disk. On startup, if a checkpoint exists, load it and resume.
The checkpoint needs four things, not just the weights:
- The model weights โ the same
float32 array the existing save_model writes.
- The optimizer state โ Adam's
m and v buffers. Without these, resuming is not a true continuation: the optimizer restarts from zero momentum and the loss spikes.
- The step number โ so the LR schedule and the log pick up where they left off.
- The tokenizer merges โ already saved as
model.tok.json, but the checkpoint should write it too so a resumed run uses the same tokenizer.
import struct
from pathlib import Path
def save_checkpoint(path, params, m, v, step, loss, tok):
"""Atomic write: tmp file + rename, so a crash mid-write doesn't corrupt."""
tmp = path.with_suffix(path.suffix + ".tmp")
with tmp.open("wb") as f:
f.write(struct.pack("<IIIf", step, len(m), len(v), loss))
for p in all_params(params):
f.write(struct.pack("<f", p.data))
for x in m:
f.write(struct.pack("<f", x))
for x in v:
f.write(struct.pack("<f", x))
tmp.replace(path)
tok.save(path.with_suffix(".tok.json"))
def load_checkpoint(path, params, tok):
"""Returns (step, m, v, loss) or None if no checkpoint exists."""
if not path.exists():
return None
with path.open("rb") as f:
step, n_m, n_v, loss = struct.unpack("<IIIf", f.read(16))
for p in all_params(params):
p.data = struct.unpack("<f", f.read(4))[0]
m = [struct.unpack("<f", f.read(4))[0] for _ in range(n_m)]
v = [struct.unpack("<f", f.read(4))[0] for _ in range(n_v)]
return step, m, v, loss
Atomic write via tmp.replace(path) โ if the process dies mid-write, the original checkpoint is untouched. Writing directly to path would corrupt it and you'd lose the last good checkpoint too. This is a one-line habit that saves runs.
Why save m and v? Adam's update is m_hat / sqrt(v_hat), and m/v are running averages. Resuming without them means the first few hundred steps after resume use wrong per-parameter scales โ the loss visibly spikes and takes a while to recover. Real training frameworks always checkpoint optimizer state.
Why save the step? The LR schedule is a function of step. Resuming at step 2,500 with a fresh step=1 would restart the warmup and re-ramp the LR โ catastrophic. The step is the schedule's only input.
What about the tokenizer? The checkpoint writes model.tok.json next to it. On resume, load_checkpoint uses the same tok object the run started with โ but the file is there so a fresh process can rebuild the tokenizer before calling load_checkpoint. The full train.py listing below shows the order.
๐
The resume test
Run python train.py corpus/, let it reach step ~600, Ctrl-C. Run it again. It should print Resuming from step 500 (loss 5.42) and continue from there โ not from step 1. The loss at step 501 should be close to the loss at step 500, not spiking. If it spikes, your optimizer state wasn't saved correctly.
โ
To run it
Requires: Python 3.9+. No pip install needed. No dependencies beyond the standard library.
Layout: seven files in one folder, plus a corpus/ directory of text files. Every file is listed in full below. train.py writes model.bin, model.tok.json, and model.ckpt.bin; infer.py loads model.bin and model.tok.json.
# 1. get a corpus (TinyStories slice, Gutenberg, or any .md/.txt files)
mkdir corpus && cp your_texts/*.txt corpus/
python train.py corpus/
python infer.py
Expected: ~170K parameters, ~1M tokens (depends on corpus), 3,000 steps. BPE training takes ~2โ3 minutes on 5M chars; training takes ~2โ3 hours on one core, resumable via model.ckpt.bin. Peak memory ~1.8 GB.
tokenizer.py โ complete (unchanged from lesson 3)
"""
tokenizer.py - byte-level BPE, learned from a corpus.
Token ids 0-255 are raw bytes. Merge number k creates id 256 + k.
Text is first split into chunks (a word with its leading space, a number,
a punctuation run, or whitespace) and merges never cross chunk borders.
Because whitespace is kept, decode(encode(text)) == text exactly.
No external dependencies. Pure Python.
"""
import json
import re
from collections import Counter
PRETOKEN = re.compile(r" ?[^\W\d]+| ?\d+| ?[^\s\w]+|\s+(?!\S)|\s+")
def pretokenize(text):
return PRETOKEN.findall(text)
def merge_ids(ids, pair, new_id):
"""Replace every adjacent occurrence of `pair` in `ids` with `new_id`."""
out, i = [], 0
while i < len(ids):
if i < len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]:
out.append(new_id)
i += 2
else:
out.append(ids[i])
i += 1
return out
def train_bpe(text, n_merges=2000):
"""Learn a merge table from raw text. Returns a list of (id, id) pairs."""
chunks = pretokenize(text)
if not chunks:
raise ValueError("empty corpus")
freq = Counter(tuple(c.encode("utf-8")) for c in chunks)
merges = []
for _ in range(n_merges):
pairs = Counter()
for word, count in freq.items():
for pair in zip(word, word[1:]):
pairs[pair] += count
if not pairs:
break
best = max(pairs, key=pairs.get)
if pairs[best] < 2:
break
new_id = 256 + len(merges)
merges.append(best)
new_freq = Counter()
for word, count in freq.items():
new_freq[tuple(merge_ids(word, best, new_id))] += count
freq = new_freq
return merges
class BPETokenizer:
"""A byte-level BPE tokenizer."""
def __init__(self, merges):
self.merges = [tuple(p) for p in merges]
self.ranks = {p: i for i, p in enumerate(self.merges)}
self.itos = {i: bytes([i]) for i in range(256)}
for i, (a, b) in enumerate(self.merges):
self.itos[256 + i] = self.itos[a] + self.itos[b]
self.vocab_size = len(self.itos)
self._cache = {}
def _encode_chunk(self, chunk):
if chunk in self._cache:
return self._cache[chunk]
ids = list(chunk.encode("utf-8"))
while len(ids) > 1:
best = min(zip(ids, ids[1:]), key=lambda p: self.ranks.get(p, float("inf")))
if best not in self.ranks:
break
ids = merge_ids(ids, best, 256 + self.ranks[best])
self._cache[chunk] = ids
return ids
def encode(self, text):
ids = []
for chunk in pretokenize(text):
ids.extend(self._encode_chunk(chunk))
return ids
def decode(self, ids):
raw = b"".join(self.itos[i] for i in ids)
return raw.decode("utf-8", errors="replace")
def save(self, path):
with open(path, "w", encoding="utf-8") as f:
json.dump(self.merges, f)
@classmethod
def load(cls, path):
with open(path, encoding="utf-8") as f:
return cls(json.load(f))
micrograd.py โ complete (unchanged)
"""
micrograd.py โ a tiny autograd engine.
backward() is iterative, which matters because a 2,256-way softmax
makes the graph deep enough to hit Python's recursion limit.
"""
import math
class Value:
def __init__(self, data, children=(), op=""):
self.data = data
self.grad = 0.0
self._prev = children
self._op = op
self._backward = lambda: None
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), "+")
def _backward():
self.grad += out.grad
other.grad += out.grad
out._backward = _backward
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), "*")
def _backward():
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = _backward
return out
def __pow__(self, n):
out = Value(self.data ** n, (self,), f"**{n}")
def _backward():
self.grad += n * (self.data ** (n - 1)) * out.grad
out._backward = _backward
return out
def exp(self):
out = Value(math.exp(self.data), (self,), "exp")
def _backward():
self.grad += out.data * out.grad
out._backward = _backward
return out
def log(self):
x = max(self.data, 1e-30)
out = Value(math.log(x), (self,), "log")
def _backward():
self.grad += (1.0 / x) * out.grad
out._backward = _backward
return out
def tanh(self):
out = Value(math.tanh(self.data), (self,), "tanh")
def _backward():
self.grad += (1.0 - out.data ** 2) * out.grad
out._backward = _backward
return out
def backward(self):
topo, visited = [], set()
stack = [(self, False)]
while stack:
v, expanded = stack.pop()
if expanded:
topo.append(v)
continue
if v in visited:
continue
visited.add(v)
stack.append((v, True))
for child in v._prev:
if child not in visited:
stack.append((child, False))
self.grad = 1.0
for v in reversed(topo):
v._backward()
v._backward = lambda: None
v._prev = ()
topo.clear(); visited.clear()
def __radd__(self, other): return self + other
def __rmul__(self, other): return self * other
def __neg__(self): return self * -1
def __sub__(self, other): return self + (-other)
def __truediv__(self, other):
other = other if isinstance(other, Value) else Value(other)
return self * (other ** -1)
def __rsub__(self, other): return (-self) + other
def __rtruediv__(self, other): return Value(other) * (self ** -1)
Value.__rsub__ = __rsub__
Value.__rtruediv__ = __rtruediv__
Value.__radd__ = __radd__
Value.__rmul__ = __rmul__
Value.__neg__ = __neg__
Value.__sub__ = __sub__
Value.__truediv__ = __truediv__
def dot(xs, ws):
out = Value(sum(x.data * w.data for x, w in zip(xs, ws)),
tuple(xs) + tuple(ws), "dot")
def _backward():
g = out.grad
for x, w in zip(xs, ws):
x.grad += w.data * g
w.grad += x.data * g
out._backward = _backward
return out
model.py โ complete (unchanged from lesson 3)
"""
model.py โ the tiny Transformer. Identical to the subword lesson.
CTX_LEN = 32 (subword tokens), vocab = 256 + n_merges.
all_params() flattens every matrix into individual Value objects.
"""
import math, random, struct
from micrograd import Value, dot
CTX_LEN = 32
D_MODEL = 32
N_LAYERS = 2
N_HEADS = 2
D_FF = 128
HEAD_DIM = D_MODEL // N_HEADS
def init_matrix(rows, cols, scale=0.08):
return [[Value(random.gauss(0, scale)) for _ in range(cols)] for _ in range(rows)]
def init_vector(n, scale=0.08):
return [Value(random.gauss(0, scale)) for _ in range(n)]
def init_model(vocab_size):
p = {}
p["tok_emb"] = init_matrix(vocab_size, D_MODEL)
p["pos_emb"] = init_matrix(CTX_LEN, D_MODEL)
p["blocks"] = []
for _ in range(N_LAYERS):
b = {}
b["ln1_g"] = init_vector(D_MODEL, 0.0)
b["ln1_b"] = init_vector(D_MODEL, 0.0)
b["wq"] = init_matrix(D_MODEL, D_MODEL)
b["wk"] = init_matrix(D_MODEL, D_MODEL)
b["wv"] = init_matrix(D_MODEL, D_MODEL)
b["wo"] = init_matrix(D_MODEL, D_MODEL)
b["ln2_g"] = init_vector(D_MODEL, 0.0)
b["ln2_b"] = init_vector(D_MODEL, 0.0)
b["w1"] = init_matrix(D_FF, D_MODEL)
b["b1"] = init_vector(D_FF, 0.0)
b["w2"] = init_matrix(D_MODEL, D_FF)
b["b2"] = init_vector(D_MODEL, 0.0)
p["blocks"].append(b)
p["lnf_g"] = init_vector(D_MODEL, 0.0)
p["lnf_b"] = init_vector(D_MODEL, 0.0)
p["head"] = init_matrix(vocab_size, D_MODEL)
return p
def _flatten(obj):
if isinstance(obj, Value):
yield obj
else:
for x in obj:
yield from _flatten(x)
def all_params(p):
for key in ("tok_emb", "pos_emb"):
yield from _flatten(p[key])
for b in p["blocks"]:
for key in ("ln1_g","ln1_b","wq","wk","wv","wo","ln2_g","ln2_b","w1","b1","w2","b2"):
yield from _flatten(b[key])
yield from _flatten(p["lnf_g"])
yield from _flatten(p["lnf_b"])
yield from _flatten(p["head"])
def matmul(x, W):
return [dot(x, row) for row in W]
def add_rows(a, b):
return [x + y for x, y in zip(a, b)]
def softmax(logits):
m = max(v.data for v in logits)
e = [(v - m).exp() for v in logits]
s = sum(v.data for v in e)
return [v / s for v in e]
def layernorm(x, g, b, eps=1e-5):
n = len(x)
mean = sum(v.data for v in x) / n
var = sum((v.data - mean) ** 2 for v in x) / n
inv = 1.0 / math.sqrt(var + eps)
out = []
for i, v in enumerate(x):
out.append((v - mean) * inv * g[i] + b[i])
return out
def gelu_approx(x):
return 0.5 * x * (1.0 + (math.sqrt(2.0 / math.pi) * (x + 0.044715 * (x ** 3))).tanh())
def attention(x, b):
Q = [matmul(t, b["wq"]) for t in x]
K = [matmul(t, b["wk"]) for t in x]
V = [matmul(t, b["wv"]) for t in x]
out = []
for i in range(CTX_LEN):
heads_out = []
for h in range(N_HEADS):
off = h * HEAD_DIM
scores = []
for j in range(i + 1):
s = sum(Q[i][off + d] * K[j][off + d] for d in range(HEAD_DIM))
scores.append(s / math.sqrt(HEAD_DIM))
probs = softmax(scores)
head = [Value(0.0) for _ in range(HEAD_DIM)]
for j in range(i + 1):
for d in range(HEAD_DIM):
head[d] = head[d] + probs[j] * V[j][off + d]
heads_out.extend(head)
out.append(matmul(heads_out, b["wo"]))
return out
def block(x, b):
a = [layernorm(t, b["ln1_g"], b["ln1_b"]) for t in x]
a = attention(a, b)
x = [add_rows(xi, ai) for xi, ai in zip(x, a)]
a = [layernorm(t, b["ln2_g"], b["ln2_b"]) for t in x]
ff = []
for t in a:
h = matmul(t, b["w1"])
h = [h[k] + b["b1"][k] for k in range(D_FF)]
h = [gelu_approx(v) for v in h]
o = matmul(h, b["w2"])
o = [o[k] + b["b2"][k] for k in range(D_MODEL)]
ff.append(o)
return [add_rows(xi, fi) for xi, fi in zip(x, ff)]
def forward(tokens, p):
x = [add_rows(p["tok_emb"][t], p["pos_emb"][i]) for i, t in enumerate(tokens)]
for b in p["blocks"]:
x = block(x, b)
x = [layernorm(t, p["lnf_g"], p["lnf_b"]) for t in x]
logits = [matmul(t, p["head"]) for t in x]
return logits
def save_model(path, p, itos, steps, loss):
with open(path, "wb") as f:
f.write(struct.pack("<III", steps, len(itos), len(list(all_params(p)))))
f.write(struct.pack("<f", loss))
for v in all_params(p):
f.write(struct.pack("<f", v.data))
def load_model(path):
with open(path, "rb") as f:
steps, vocab_size, n_params = struct.unpack("<III", f.read(12))
loss, = struct.unpack("<f", f.read(4))
params = []
for _ in range(n_params):
params.append(Value(struct.unpack("<f", f.read(4))[0]))
return steps, vocab_size, loss, params
data.py โ complete
"""
data.py โ corpus loading, tokenization, and the train/val split.
"""
from pathlib import Path
from tokenizer import train_bpe, BPETokenizer
TEXT_EXTS = {".txt", ".md", ".rst"}
VAL_FRACTION = 0.1
N_MERGES = 2000
def load_corpus(input_dir):
"""Read every text file under input_dir, concatenate with blank lines."""
src = Path(input_dir)
if not src.exists():
raise SystemExit(f"input dir not found: {src}")
files = sorted(p for p in src.rglob("*")
if p.is_file() and p.suffix.lower() in TEXT_EXTS)
if not files:
raise SystemExit(f"no text files under {src}")
parts = []
for p in files:
try:
t = p.read_text(encoding="utf-8")
except UnicodeDecodeError:
t = p.read_text(encoding="latin-1")
t = t.strip()
if t:
parts.append(t)
return "\n\n".join(parts)
def build_dataset(input_dir, n_merges=N_MERGES, val_fraction=VAL_FRACTION):
"""Returns (train_tokens, val_tokens, tokenizer)."""
text = load_corpus(input_dir)
print(f"Corpus: {len(text):,} chars")
print(f"Training BPE ({n_merges} merges)...")
merges = train_bpe(text, n_merges)
tok = BPETokenizer(merges)
print(f"Vocabulary: {tok.vocab_size} tokens")
all_ids = tok.encode(text)
n_val = int(len(all_ids) * val_fraction)
n_train = len(all_ids) - n_val
train_ids = all_ids[:n_train]
val_ids = all_ids[n_train:]
print(f"Tokens: {len(all_ids):,} total ({n_train:,} train / {n_val:,} val)")
return train_ids, val_ids, tok
train.py โ complete
"""train.py โ pretrain the subword model on real text."""
import sys, time, random, gc, math, struct
from pathlib import Path
from tokenizer import BPETokenizer
from micrograd import Value
from model import CTX_LEN, init_model, all_params, forward, softmax, save_model
from data import build_dataset
BATCH_SIZE = 4
ACCUM_STEPS = 8
STEPS = 3000
LR_MAX = 3e-3
LR_MIN = 3e-4
WARMUP = 100
EVAL_EVERY = 100
CHECKPOINT_EVERY = 500
EVAL_BATCHES = 20
N_MERGES = 2000
def compute_loss(logits, targets):
total = Value(0.0)
for t in range(len(logits)):
probs = softmax(logits[t])
total = total - probs[targets[t]].log()
return total / len(logits)
def lr_at(step):
if step <= WARMUP:
return LR_MAX * step / WARMUP
progress = (step - WARMUP) / max(1, STEPS - WARMUP)
return LR_MIN + 0.5 * (LR_MAX - LR_MIN) * (1 + math.cos(math.pi * progress))
def sample_batch(ids):
x, y = [], []
for _ in range(BATCH_SIZE):
j = random.randint(0, len(ids) - CTX_LEN - 1)
x.append(ids[j : j + CTX_LEN])
y.append(ids[j + 1 : j + CTX_LEN + 1])
return x, y
def evaluate(params, val_ids):
total = 0.0
for _ in range(EVAL_BATCHES):
x, y = sample_batch(val_ids)
for xi, yi in zip(x, y):
logits = forward(xi, params)
total += compute_loss(logits, yi).data
return total / (EVAL_BATCHES * BATCH_SIZE)
def save_checkpoint(path, params, m, v, step, loss, tok):
tmp = path.with_suffix(path.suffix + ".tmp")
with tmp.open("wb") as f:
f.write(struct.pack("<IIIf", step, len(m), len(v), loss))
for p in all_params(params):
f.write(struct.pack("<f", p.data))
for x in m:
f.write(struct.pack("<f", x))
for x in v:
f.write(struct.pack("<f", x))
tmp.replace(path)
tok.save(path.with_suffix(".tok.json"))
def load_checkpoint(path, params, tok):
if not path.exists():
return None
with path.open("rb") as f:
step, n_m, n_v, loss = struct.unpack("<IIIf", f.read(16))
for p in all_params(params):
p.data = struct.unpack("<f", f.read(4))[0]
m = [struct.unpack("<f", f.read(4))[0] for _ in range(n_m)]
v = [struct.unpack("<f", f.read(4))[0] for _ in range(n_v)]
return step, m, v, loss
def main():
random.seed(0)
gc.disable()
input_dir = Path(sys.argv[1] if len(sys.argv) > 1 else "corpus")
out_path = Path(sys.argv[2] if len(sys.argv) > 2 else "model.bin")
ckpt_path = out_path.with_suffix(".ckpt.bin")
train_ids, val_ids, tok = build_dataset(input_dir, n_merges=N_MERGES)
if len(train_ids) <= CTX_LEN + 1 or len(val_ids) <= CTX_LEN + 1:
raise SystemExit("Corpus too short after split")
vocab_size = tok.vocab_size
params = init_model(vocab_size)
plist = list(all_params(params))
print(f"Params: {len(plist):,}")
m = [0.0] * len(plist)
v = [0.0] * len(plist)
b1, b2, eps = 0.9, 0.999, 1e-8
start_step = 0
ckpt = load_checkpoint(ckpt_path, params, tok)
if ckpt is not None:
start_step, m, v, last_loss = ckpt
print(f"Resuming from step {start_step} (loss {last_loss:.4f})")
t0 = time.time()
recent_train = []
print(f"Training {start_step+1}..{STEPS} (warmup {WARMUP}, LR_MAX {LR_MAX}, accum {ACCUM_STEPS})")
for step in range(start_step + 1, STEPS + 1):
lr = lr_at(step)
step_loss = Value(0.0)
for _ in range(ACCUM_STEPS):
x, y = sample_batch(train_ids)
micro = Value(0.0)
for xi, yi in zip(x, y):
micro = micro + compute_loss(forward(xi, params), yi)
micro = micro / BATCH_SIZE
step_loss = step_loss + micro / ACCUM_STEPS
micro.backward()
for i, p in enumerate(plist):
g = p.grad
m[i] = b1 * m[i] + (1 - b1) * g
v[i] = b2 * v[i] + (1 - b2) * g * g
m_hat = m[i] / (1 - b1 ** step)
v_hat = v[i] / (1 - b2 ** step)
p.data -= lr * m_hat / (v_hat ** 0.5 + eps)
p.grad = 0.0
recent_train.append(step_loss.data)
if step % EVAL_EVERY == 0 or step == STEPS:
train_avg = sum(recent_train[-EVAL_EVERY:]) / len(recent_train[-EVAL_EVERY:])
val_loss = evaluate(params, val_ids)
print(f"step {step:5d} lr {lr:.2e} train {train_avg:.4f} val {val_loss:.4f}")
if step % CHECKPOINT_EVERY == 0 or step == STEPS:
save_checkpoint(ckpt_path, params, m, v, step, step_loss.data, tok)
print(f" checkpoint saved at step {step}")
final_val = evaluate(params, val_ids)
save_model(out_path, params, tok.itos, STEPS, final_val)
tok.save(out_path.with_suffix(".tok.json"))
print(f"Saved: {out_path} (final val loss: {final_val:.4f})")
print(f"Done in {(time.time() - t0)/60:.1f} min")
if __name__ == "__main__":
main()
infer.py โ complete (unchanged from lesson 3)
"""infer.py โ sample from the subword model."""
import sys, struct, math, random
from pathlib import Path
from tokenizer import BPETokenizer
from micrograd import Value
from model import CTX_LEN, init_model, all_params, forward, softmax
def load_weights_into(model, flat):
it = iter(flat)
for v in all_params(model):
v.data = next(it)
def load_model_bin(path):
with open(path, "rb") as f:
steps, vocab_size, n_params = struct.unpack("<III", f.read(12))
loss, = struct.unpack("<f", f.read(4))
flat = [struct.unpack("<f", f.read(4))[0] for _ in range(n_params)]
model = init_model(vocab_size)
load_weights_into(model, flat)
return model, steps, vocab_size, loss
def sample(logits, temperature=1.0, top_k=None):
if temperature <= 0:
return max(range(len(logits)), key=lambda i: logits[i].data)
scaled = [v / temperature for v in logits]
probs = softmax(scaled)
if top_k:
idx = sorted(range(len(probs)), key=lambda i: probs[i].data, reverse=True)[:top_k]
keep = {i: probs[i].data for i in idx}
z = sum(keep.values())
r = random.random() * z
acc = 0.0
for i, p in keep.items():
acc += p
if r <= acc:
return i
return idx[-1]
r = random.random()
acc = 0.0
for i, p in enumerate(probs):
acc += p.data
if r <= acc:
return i
return len(probs) - 1
def _pad_to_ctx(tokens):
if len(tokens) >= CTX_LEN:
return tokens[-CTX_LEN:]
return [32] * (CTX_LEN - len(tokens)) + tokens
def generate(model, tok, prompt, n_tokens=60, temperature=1.0, top_k=None):
tokens = tok.encode(prompt)
if not tokens:
tokens = [32]
for _ in range(n_tokens):
ctx = _pad_to_ctx(tokens)
logits = forward(ctx, model)
next_id = sample(logits[-1], temperature, top_k)
tokens.append(next_id)
return tok.decode(tokens)
def main():
random.seed(0)
model_bin = Path(sys.argv[1] if len(sys.argv) > 1 else "model.bin")
tok_json = Path(sys.argv[2] if len(sys.argv) > 2 else "model.tok.json")
if not model_bin.exists():
raise SystemExit(f"missing {model_bin}; run train.py first")
if not tok_json.exists():
raise SystemExit(f"missing {tok_json}; run train.py first")
model, steps, vocab_size, loss = load_model_bin(model_bin)
tok = BPETokenizer.load(tok_json)
if tok.vocab_size != vocab_size:
raise SystemExit(f"vocab mismatch: model={vocab_size} tokenizer={tok.vocab_size}")
print(f"Loaded: {steps} steps, vocab {vocab_size}, val loss {loss:.4f}")
print("Commands: :t <temp> :k <top_k> :q (or type a prompt)")
temp, topk = 1.0, None
while True:
try:
line = input("> ").strip()
except (EOFError, KeyboardInterrupt):
print()
break
if not line:
continue
if line == ":q":
break
if line.startswith(":t "):
temp = float(line[3:]); print(f"temperature = {temp}"); continue
if line.startswith(":k "):
topk = int(line[3:]); print(f"top_k = {topk}"); continue
out = generate(model, tok, line, n_tokens=60, temperature=temp, top_k=topk)
print(out)
if __name__ == "__main__":
main()
๐งฐ
Troubleshooting
Smoke test first โ set STEPS = 20, N_MERGES = 100, and point at a tiny corpus. Everything should finish in under a minute and produce model.bin + model.tok.json + model.ckpt.bin.
"Corpus too short after split" โ your corpus is under ~10K tokens. Add files or lower VAL_FRACTION.
Loss spikes at step 1 โ LR_MAX is too high for your model. Try 1e-3 and increase WARMUP to 200.
Resume causes a loss spike โ optimizer state wasn't loaded. Check that the log says "Resuming from step โฆ".
Val loss goes up โ overfitting. Stop earlier, get more data, or reduce N_LAYERS from 2 to 1 as an experiment.
'list' object has no attribute 'grad' โ all_params is yielding matrix rows. Use the full model.py listing above, which flattens via _flatten.
Training is very slow โ pure Python, ~1โ2 s/step with ACCUM_STEPS=8. Lower ACCUM_STEPS to 2โ4 to iterate faster; raise it back for the real run.
Checkpoint file is huge โ 3 arrays of ~170K float32 โ 2 MB. That's expected. Delete model.ckpt.bin when a run finishes to save space.