A production framework does the same thing, but with GPU tensors and C-level loops. The math is identical. Here every Value is a Python object (about 500 bytes), so time and memory scale with the number of graph nodes. That is why the lesson shrinks the model to ~33K parameters and fuses each dot product into one node. A naive version that creates a separate node for every multiply and add ran out of memory on a 4 GB machine after just 16 tokens, even for a 64-wide model.
Build a tiny AI from scratch
Seven short steps to build a working character-level Transformer using only the Python standard library. No frameworks, no tensor libraries, no dependencies. Every gradient, every matrix multiply, every update is written out β so you can see what a neural network actually is.
A tiny Transformer
2 blocks, 2 heads, 32-wide embeddings, ~33K parameters β a scaled-down GPT. Same architecture, small enough to train in pure Python.
Learns from text
Reads corpus.txt, predicts the next character. Loss drops from ~4.7 β ~2.7 over 300 steps (perplexity β105 β β14). Expect word-shaped fragments, not sentences.
Five runnable files
micrograd.py, model.py, train.py, infer.py, build_corpus.py. No dependencies beyond Python itself.
Build a tiny autograd engine
A neural network needs two things: run operations on big collections of numbers, and compute derivatives automatically. We're going to build both from scratch. Step 1 builds the derivative part β an autograd engine in ~100 lines.
The core idea: every value is stored in a small object called a Value. It remembers the numbers that produced it, so it can later compute "how much did I contribute to the final loss?" That's what backpropagation is.
import math
class Value:
"""A number that remembers how it was made."""
def __init__(self, data, children=(), op=""):
self.data = data
self.grad = 0.0 # d(output)/d(this value)
self._prev = children # values that produced this one
self._op = op
self._backward = lambda: None
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), "+")
def _backward():
self.grad += out.grad # d(a+b)/da = 1
other.grad += out.grad # d(a+b)/db = 1
out._backward = _backward
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), "*")
def _backward():
self.grad += other.data * out.grad # d(ab)/da = b
other.grad += self.data * out.grad # d(ab)/db = a
out._backward = _backward
return out
def __pow__(self, n):
out = Value(self.data ** n, (self,), f"**{n}")
def _backward():
self.grad += n * (self.data ** (n - 1)) * out.grad
out._backward = _backward
return out
def exp(self):
out = Value(math.exp(self.data), (self,), "exp")
def _backward():
self.grad += out.data * out.grad # d(e^x)/dx = e^x
out._backward = _backward
return out
def log(self):
x = max(self.data, 1e-30) # guard against log(0)
out = Value(math.log(x), (self,), "log")
def _backward():
self.grad += (1.0 / x) * out.grad
out._backward = _backward
return out
def tanh(self):
out = Value(math.tanh(self.data), (self,), "tanh")
def _backward():
self.grad += (1.0 - out.data ** 2) * out.grad
out._backward = _backward
return out
def backward(self):
"""Compute gradients for every Value that produced this one."""
topo, visited = [], set()
def build(v):
if v not in visited:
visited.add(v)
for child in v._prev:
build(child)
topo.append(v)
build(self)
self.grad = 1.0 # d(output)/d(output) = 1
for v in reversed(topo):
v._backward()
v._backward = lambda: None # free the graph as we go: breaking the
v._prev = () # closure cycles lets Python reclaim memory
# immediately (one backward() per graph)
topo.clear(); visited.clear() # `build` refers to itself, so it would keep the graph alive
# helper operators so `2 * v`, `v - 1` and `v / 2` work
def __radd__(self, other): return self + other
def __rmul__(self, other): return self * other
def __neg__(self): return self * -1
def __sub__(self, other): return self + (-other)
def __truediv__(self, other):
other = other if isinstance(other, Value) else Value(other)
return self * (other ** -1)
Value.__radd__ = __radd__
Value.__rmul__ = __rmul__
Value.__neg__ = __neg__
Value.__sub__ = __sub__
Value.__truediv__ = __truediv__
def dot(xs, ws):
"""Dot product as ONE graph node (instead of 2n tiny + and * nodes).
This single trick makes the model ~100x cheaper in memory and time."""
out = Value(sum(x.data * w.data for x, w in zip(xs, ws)),
tuple(xs) + tuple(ws), "dot")
def _backward():
g = out.grad
for x, w in zip(xs, ws):
x.grad += w.data * g # d(xΒ·w)/dx = w
w.grad += x.data * g # d(xΒ·w)/dw = x
out._backward = _backward
return out
That's the entire autograd engine. Every arithmetic operation returns a new Value that stores two things: the answer (.data), and a small closure that knows how to send the gradient back to the inputs (._backward).
When you call .backward(), the engine walks the computation graph in reverse topological order and applies each ._backward in turn. That's backpropagation. There is nothing else.
The last function, dot, is a speed trick rather than new math: it computes a whole dot product as one graph node with one backward closure, instead of one multiply node and one add node per term. Everything in Step 3 is built on it. Two practical notes: backward() frees the graph as it walks it (so call it once per forward pass, and inspect the graph before calling it), and it clears its own helper state so Python can reclaim the memory immediately.
Test it: a = Value(2); b = Value(3); c = a * b; c.backward() β then check a.grad == 3 and b.grad == 2. Chain rule, done.
Define the model's shape
Same architecture as a modern language model, just smaller. Every constant is deliberately chosen so the model fits in a few tens of thousands of parameters β small enough that pure-Python training finishes in tens of minutes.
CTX_LEN = 32 # characters visible at once
D_MODEL = 32 # width of each character's vector
N_LAYERS = 2 # Transformer blocks, stacked
N_HEADS = 2 # attention heads per block
D_FF = 128 # FFN inner width (4x D_MODEL)
HEAD_DIM = D_MODEL // N_HEADS # 16 dims per head
# ---- how training runs ----
BATCH_SIZE = 4 # windows per step
STEPS = 300 # total update steps
LR = 3e-3 # Adam learning rate
The vocab depends on the corpus. For a typical 1M-character text, vocab is ~90β120 characters (plus <unk>). For the reference corpus in this lesson (467K characters of Pythonβs built-in documentation text), vocab = 105 including <unk>. That gives 32,640 parameters, about 33K. Run train.py and it prints the exact count. (The embedding table and the output head each add D_MODEL parameters per vocabulary entry.)
The math of one forward pass
Every symbol is one line in the code. Shapes are for one window with T = 64 tokens.
for each block:
a = LayerNorm(h) (T, D_MODEL)
q, k, v = split(a @ W_qkv) (T, D_MODEL) each
A = softmax(qΒ·kα΅ / βHEAD_DIM, causal), per head (T, T)
h = h + concat(A @ v over heads) @ W_out (T, D_MODEL)
h = h + FFN(LayerNorm(h)) (T, D_MODEL)
logits = LayerNorm(h) @ W_head (T, vocab)
Every @ is a matrix multiply built from dot products in Step 3. Nothing hidden.
The only difference from a full-size model is size: D_MODEL=32 instead of 768, N_LAYERS=2 instead of 12, CTX_LEN=32 instead of 1024. The core math is the same (GPT-2 also adds biases to its linear layers and ties its input and output embeddings). Everything you learn here transfers directly to the full-size version.
Forward pass β every operation
A Transformerβs forward pass is built from five primitives (plus embedding lookups): matmul, add, softmax, LayerNorm, and activation. Each one is a short function below, using only lists of Value objects.
from micrograd import Value, dot
import math, random
# ---- primitives ----
def matmul(X, W):
"""X: list of rows, W: list of rows. Returns X @ W."""
cols = list(zip(*W)) # columns of W
return [[dot(x_row, col) for col in cols] for x_row in X]
def add_rows(X, Y):
return [[a + b for a, b in zip(rx, ry)] for rx, ry in zip(X, Y)]
def softmax(row):
m = max(v.data for v in row) # subtract max for stability
shifted = [v - m for v in row]
exps = [v.exp() for v in shifted]
total = sum(exps, Value(0.0))
return [v / total for v in exps]
def layernorm(row, gamma, beta, eps=1e-5):
n = len(row)
mean = sum(row, Value(0.0)) / n
diffs = [v - mean for v in row]
var = sum((d * d for d in diffs), Value(0.0)) / n
denom = (var + eps) ** 0.5
norm = [d / denom for d in diffs]
return [g * x + b for g, x, b in zip(gamma, norm, beta)]
def gelu_approx(x):
c = 0.7978845608 # sqrt(2/pi)
return 0.5 * x * (1.0 + ((x + 0.044715 * x * x * x) * c).tanh())
Why these five? Nearly every operation here decomposes into them. A QKV projection is a single matmul (this model has no biases). An attention score is a matmul plus a divide plus a softmax. LayerNorm is a mean plus a variance plus a rescale. The FFN is two matmuls separated by a GELU.
Why matmul uses dot: each output element is one dot product of a row of X with a column of W, so it becomes a single graph node. That keeps the graph small enough to fit in memory and is the reason this model trains at all in pure Python.
Why lists of Value? Because Value remembers how it was made, so calling .backward() on the final loss automatically flows gradients back through every primitive β including through every dot product. You write the forward pass; the gradients are free.
The Transformer block
Now compose the primitives into a block. Two sublayers, each with a residual connection.
# ---- attention + block ----
def attention(h, W_qkv, W_out):
T = len(h)
qkv = matmul(h, W_qkv) # (T, 3 * D_MODEL)
scale = 1.0 / math.sqrt(HEAD_DIM)
out = [[] for _ in range(T)]
for head in range(N_HEADS):
lo = head * HEAD_DIM
q = [r[lo : lo + HEAD_DIM] for r in qkv]
k = [r[D_MODEL + lo : D_MODEL + lo + HEAD_DIM] for r in qkv]
v = [r[2 * D_MODEL + lo : 2 * D_MODEL + lo + HEAD_DIM] for r in qkv]
for t in range(T):
# causal mask: position t only looks at positions 0..t
scores = [dot(q[t], k[s]) * scale for s in range(t + 1)]
weights = softmax(scores)
for d in range(HEAD_DIM):
out[t].append(dot(weights, [v[s][d] for s in range(t + 1)]))
return matmul(out, W_out) # heads are concatenated in `out`
def block(h, p, i):
# 1. attention sublayer
a = [layernorm(row, p[f"ln1_g_{i}"], p[f"ln1_b_{i}"]) for row in h]
h = add_rows(h, attention(a, p[f"W_qkv_{i}"], p[f"W_out_{i}"]))
# 2. feed-forward sublayer
a = [layernorm(row, p[f"ln2_g_{i}"], p[f"ln2_b_{i}"]) for row in h]
ff = matmul(a, p[f"W_ff1_{i}"]) # (T, D_FF)
ff = [[gelu_approx(x) for x in row] for row in ff]
ff = matmul(ff, p[f"W_ff2_{i}"]) # (T, D_MODEL)
return add_rows(h, ff)
def forward(tokens, p):
h = [[p["tok_embed"][tok][d] + p["pos_embed"][t][d] for d in range(D_MODEL)]
for t, tok in enumerate(tokens)]
for i in range(N_LAYERS):
h = block(h, p, i)
h = [layernorm(row, p["ln_f_g"], p["ln_f_b"]) for row in h]
return matmul(h, p["W_head"]) # (T, vocab)
The full forward pass is: embed β block β block β final LayerNorm β linear projection to vocab. That's four lines of code, given the functions above.
h = block(h, p, 0)
h = block(h, p, 1)
h = [layernorm(row, p["ln_f_g"], p["ln_f_b"]) for row in h]
logits = matmul(h, p["W_head"]) (T, vocab)
With N_HEADS=2 and D_MODEL=32, each head works on its own 16-dim slice of q, k and v, with its own attention weights. The two headsβ outputs are concatenated and mixed by W_out. The causal mask is implemented by only scoring positions 0..t for position t, so future characters never enter the softmax.
Backward pass β free gradients
Here's the beautiful part: you don't write the backward pass. The autograd engine from Step 1 does it. You just call .backward() on the loss.
def compute_loss(logits, targets):
"""Cross-entropy loss. logits: (T, vocab), targets: (T,)."""
total = Value(0.0)
for t in range(len(logits)):
probs = softmax(logits[t])
total = total - probs[targets[t]].log()
return total / len(logits)
# one full training step (plain gradient descent for now; Step 5 upgrades it to Adam)
logits = forward(x, params) # (T, vocab) list of Values
loss = compute_loss(logits, y)
loss.backward() # <- everything happens here
for p in all_params(params):
p.data -= LR * p.grad # nudge each weight downhill
p.grad = 0.0 # reset for next step
That's the entire backward pass. When you call loss.backward(), the autograd engine walks the computation graph from Step 3, and at every Value, executes the small closure that was registered when the value was created. The gradients flow backward through matmuls, through softmax, through LayerNorm, through GELU β all automatically.
The only place you need to be careful is that p.grad accumulates across backward calls. That's why you reset to 0.0 after each update. Also remember that backward() frees the graph as it goes, so call it once per forward pass.
Every operation in the forward pass is a Value that knows how its output changes when each input changes. Backprop is just the chain rule applied in reverse topological order. The autograd engine collects the results. No manual gradient derivation required.
Train β the loop
Every training step is the same four operations: sample, forward, backward, update. The update rule is written out explicitly.
# Adam optimizer state (one entry per parameter)
plist = list(all_params(params))
m = [0.0] * len(plist) # running mean of gradients
v = [0.0] * len(plist) # running mean of squared gradients
b1, b2, eps = 0.9, 0.999, 1e-8
for step in range(1, STEPS + 1):
# 1. sample a batch of windows, 2. forward pass on each
loss = Value(0.0)
for _ in range(BATCH_SIZE):
j = random.randint(0, len(data) - CTX_LEN - 1)
x = data[j : j + CTX_LEN]
y = data[j + 1 : j + CTX_LEN + 1]
loss = loss + compute_loss(forward(x, params), y)
loss = loss / BATCH_SIZE
# 3. backward pass - the whole point of Step 1
loss.backward()
# 4. Adam update (written out explicitly)
for i, p in enumerate(plist):
m[i] = b1 * m[i] + (1 - b1) * p.grad
v[i] = b2 * v[i] + (1 - b2) * p.grad ** 2
m_hat = m[i] / (1 - b1 ** step)
v_hat = v[i] / (1 - b2 ** step)
p.data -= LR * m_hat / (v_hat ** 0.5 + eps)
p.grad = 0.0
if step == 1 or step % 50 == 0:
print(f"step {step:4d} loss {loss.data:.4f}")
What you can see that a framework hides:
β The optimizer state. Adam keeps two running averages per parameter (m and v). A framework hides these inside a built-in optimizer object; here they're plain lists.
β‘ The bias correction. m_hat and v_hat are divided by 1 - b^step to correct for the fact that at step 1 the running averages are still close to zero. Without it, the first few updates would be far too small.
β’ The reset. p.grad = 0.0 after each update. Forget this and gradients accumulate across steps, so each update uses a pile of stale gradients and training goes wrong.
β£ The garbage collector. train.py calls gc.disable(). Each step allocates millions of small objects that reference each other; Pythonβs cycle collector kept rescanning them, and in my test the forward pass was about 10Γ slower with it on. Because backward() frees the graph as it goes, memory stays flat without it.
What you'll see
Loss trajectory from a real run on 467K characters of Pythonβs built-in documentation text (step 250 omitted; each line after the first is the average of the previous 50 steps):
Vocab: 105 Tokens: 467,243
Params: 32,640
Training for 300 steps...
step 1 loss 4.6803
step 50 loss 3.6219
step 100 loss 3.0556
step 150 loss 2.8065
step 200 loss 2.7181
step 300 loss 2.6631
Saved: model.bin (avg loss over last 50 steps: 2.6631)
Done in 1143.2s
Loss falls from about 4.7 (random guessing is ln 105 β 4.65) to about 2.7: perplexity drops from β105 to β14. The run only sees ~38K of the corpusβs 467K characters (300 steps Γ 4 windows Γ 32), so nearly every batch is new text and training loss is a fair stand-in for held-out loss. It took about 19 minutes in my test container; yours will differ, and the script prints the time. The curve is still falling at step 300, so more steps help β this run is short on purpose.
Save and load
With no framework, there's no built-in save/load. We write a simple binary format: a small header, then all the weights as 32-bit floats. That's it β that's all a checkpoint is.
import struct
SHAPE = (CTX_LEN, D_MODEL, N_LAYERS, N_HEADS, D_FF)
def save_model(path, params, chars, step, loss):
with open(path, "wb") as f:
f.write(struct.pack("<5I", *SHAPE)) # so we can detect a mismatch
f.write(struct.pack("<I", len(chars)))
for c in chars:
b = c.encode("utf-8") # 1 to 4 bytes per character
f.write(struct.pack("<B", len(b)) + b) # length prefix, then bytes
f.write(struct.pack("<If", step, loss))
for p in all_params(params):
f.write(struct.pack("<f", p.data))
def load_model(path):
with open(path, "rb") as f:
if struct.unpack("<5I", f.read(20)) != SHAPE:
raise SystemExit("model.bin was trained with different constants")
n_chars = struct.unpack("<I", f.read(4))[0]
chars = []
for _ in range(n_chars):
n = f.read(1)[0]
chars.append(f.read(n).decode("utf-8"))
step, loss = struct.unpack("<If", f.read(8))
params = init_model(n_chars + 1) # +1 for <unk>
for p in all_params(params):
p.data = struct.unpack("<f", f.read(4))[0]
return params, chars, step, loss
What's happening: the file starts with the five model constants (so loading a checkpoint with the wrong shape fails loudly), then the number of vocabulary entries, then each character as a length byte followed by its UTF-8 bytes, then the training step and loss, then every parameter as a 4-byte float. Characters are length-prefixed because many (Γ©, β, β) take several bytes in UTF-8; writing one byte per character would corrupt the vocabulary.
Why this format? Because it's the smallest thing that could work. More complex formats exist for good reasons (arbitrary tensor shapes, dtypes, device placement, metadata), but none of that matters here. The 32,640 parameters take exactly 130,560 bytes; the reference model.bin is 130,816 bytes once the header and vocabulary are included. Weights are stored as 32-bit floats, so they are rounded slightly on save.
Every modern model file (safetensors, GGUF, and others) is essentially this: a header describing shapes and dtypes, followed by the raw weights. The format matters for interoperability, not for correctness. Once you've written this short saver, you understand what every checkpoint loader actually does.
Generate text
Load the weights, feed a prompt, predict the next character, append it, repeat. Every step is a forward pass β no backward, no gradients.
def generate(params, chars, prompt, n_tokens=150, temperature=0.8, top_k=40):
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for i, c in enumerate(chars)}
unk = len(chars)
tokens = [stoi.get(c, unk) for c in prompt] or [unk]
for _ in range(n_tokens):
# 1. forward pass on the last CTX_LEN tokens
gc.disable() # a forward pass allocates a huge graph;
logits = forward(tokens[-CTX_LEN:], params) # collecting mid-way is very slow
nxt = [Value(v.data) for v in logits[-1]] # keep only the numbers we need
del logits
gc.enable(); gc.collect() # nothing calls backward(), so free it here
# 2. temperature scaling
nxt = [v * (1.0 / max(temperature, 1e-6)) for v in nxt]
# 3. top-k filtering: keep only the k largest logits
if top_k > 0:
cutoff = sorted((v.data for v in nxt), reverse=True)[:top_k][-1]
nxt = [v if v.data >= cutoff else Value(-1e9) for v in nxt]
# 4. softmax -> probabilities
probs = softmax(nxt)
# 5. sample one token from the distribution
r, cumsum = random.random(), 0.0
choice = len(probs) - 1
for i, p in enumerate(probs):
cumsum += p.data
if cumsum >= r:
choice = i
break
tokens.append(choice)
return "".join(itos.get(t, "<unk>") for t in tokens)
A note on speed: generation also builds a graph, so it turns the garbage collector off during the forward pass and collects once per character. That took generation from about 2.5 s to about 0.34 s per character in my container, so 150 characters take about a minute.
Step 5 is the interesting one. A one-line sampler call does the same thing in a few characters. Here you can see the algorithm: draw a uniform random number between 0 and 1, then walk the probability distribution accumulating until you pass it. That's called inverse transform sampling, and it's what every sampler uses.
Why the loop instead of random.choices(population, weights)? Because random.choices does the same thing internally, and seeing the loop makes it clear that "sampling from a distribution" isn't magic β it's a linear scan.
A real session with the trained model. :t sets temperature and :k sets top-k, and the model is only 32,640 parameters at loss β2.7, so expect English-shaped fragments rather than words:
> :t 0.5
> :k 10
> The
The t atharexthe te are is lle s the e tiontes on ontre ase ss on thed te the ctice as le ioftins at astirertins as. sthertere the an ons ante iconcr inin
The model sees at most 32 characters β about 6 words. It can't complete a sentence, let alone follow a paragraph. This is the biggest limitation of the model, and it comes from CTX_LEN = 32 combined with character-level tokens. A longer context costs time: attention work grows with the square of the window.
That's the whole build
- Autograd β ~100 lines. Enables everything else.
- Shape β small enough to run in pure Python (~33K params).
- Forward β 5 primitives composed into a Transformer.
- Backward β free, courtesy of the autograd engine.
- Train β Adam written out explicitly.
- Save β 20-line binary format.
- Generate β sample loop, no gradient.
What is this model actually good for?
It's not a product. It's a microscope. Here's what you can genuinely do with it:
- See what autograd really is. Print any intermediate
Value, inspect its.gradafter backward. This is impossible in a framework β you'd have to add hooks. - Watch gradients flow through attention. After
loss.backward(), inspectparams["W_qkv_0"][0][0].grad(before the update loop zeroes it) and see how it responds to the loss. That chain is the entire learning mechanism. - Study the optimizer. Raise
LRfrom 3e-3 to 3e-2 and compare the loss curve, then lower it to 3e-4 and compare again. The relationship is direct because you can read the update rule. - Understand sampling. Try
:t 0.1(almost always the single most likely character, close to greedy decoding) against:t 1.5(much more random). - Understand checkpoints. The saver is short. Add a feature (a new field, a compression) and understand exactly what happens to the file.
What it can't do: anything that requires real scale. This is a teaching model, not a shipping model. But the core ideas carry over to modern language models; the biggest differences are scale and a handful of refinements (see Stage 4).
Project Gutenberg has thousands of free public-domain books. Wikipedia has dumps. Or any folder of your own .md or .txt files. Aim for at least a few hundred thousand characters (the reference run used 467K). The reference corpus is Python's own documentation text: python -c "from pydoc_data.topics import topics; open('docs/py.md','w',encoding='utf-8').write('\n\n'.join(topics.values()))"
How to scale it up
The architecture you just built is GPT-2's architecture, minus biases and tied embeddings, scaled down by ~3,800Γ in parameter count. Scaling up is a sequence of one-change-at-a-time experiments. Do them in order.
Stage 1 β bigger model, same code
The pure-Python version is ~33K parameters. Doubling D_MODEL to 64 (and D_FF to 256) gives ~114K parameters and roughly 4Γ the compute per token, so a run that took ~19 minutes would take over an hour (an estimate; I didn't time that configuration). Try it once; you'll appreciate why frameworks exist.
Stage 2 β move to a tensor library
Take the same architecture and rewrite it with a tensor library. Now you can:
- Use tensors instead of lists (often orders of magnitude faster)
- Call
.backward()on tensors instead ofValue(identical idea, faster) - Use a built-in multi-head attention layer instead of writing it out
What you learn: which parts of a framework are doing real work vs. which parts are convenience. Both are valuable to know.
Stage 3 β reproduce GPT-2 small
Same architecture as your tiny model, but at 124M parameters (~3,800Γ yours). Same code structure, different constants:
β 124M parameters Β· needs a GPU Β· ~40GB of text Β· BPE tokenizer
At this size you also need a proper tokenizer (subword, not character-level) and a learning-rate schedule. Those are the two biggest new ideas at this stage.
Stage 4 β modern block improvements
Llama and Mistral keep the same overall shape but swap four pieces inside each block:
| Old (GPT-2) | New (Llama) | Why |
|---|---|---|
| LayerNorm | RMSNorm | Cheaper, no mean subtraction |
| Learned position table | Rotary positions (RoPE) | Extrapolates to longer contexts |
| GELU | SwiGLU | Better loss per parameter |
| Full multi-head attention | Grouped-query attention (GQA) | Smaller KV cache at inference |
Stage 5 β from base model to assistant
The model you built continues text. To make it answer questions:
- Supervised fine-tuning β train on (question, answer) pairs
- Preference training (RLHF or DPO) β train on rankings
- Tool use β emit structured function calls
None of these change the architecture. They're new training objectives on top of the same base model.
Concrete first moves
- Run the five files as-is. Confirm 32,640 parameters and a final loss near 2.7 (it will vary with your corpus).
- Print the gradient of a specific weight after one training step. Understand where the number came from.
- Double
D_MODELto 64 andD_FFto 256. Retrain. Expect about 3.5Γ the parameters (~114K) and roughly 4Γ the training time. - Only then: rewrite with a tensor library and compare speed.
The full source
Requires: Python 3.9+. No pip install needed. No dependencies beyond the standard library.
Layout: all five files in one folder, plus a corpus.txt.
mkdir docs && cp some_text.md docs/ python build_corpus.py ./docs corpus.txt python train.py corpus.txt model.bin python infer.py
Expected: 32,640 parameters with the reference corpus; training took about 19 minutes in my test container (your machine will differ), then ~0.3 s per generated character. Peak memory was about 600 MB. Tested on Python 3.12.
micrograd.py
"""
micrograd.py β a tiny autograd engine.
Every operation returns a Value that knows how to push its gradient
back to the Values that produced it. Call .backward() on any Value.
Usage:
a = Value(2.0)
b = Value(3.0)
c = a * b + a
c.backward()
print(a.grad) # 4.0 (dc/da = b + 1)
print(b.grad) # 2.0 (dc/db = a)
"""
import math
class Value:
"""A number that remembers how it was made."""
def __init__(self, data, children=(), op=""):
self.data = data
self.grad = 0.0 # d(output)/d(this value)
self._prev = children # values that produced this one
self._op = op
self._backward = lambda: None
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), "+")
def _backward():
self.grad += out.grad # d(a+b)/da = 1
other.grad += out.grad # d(a+b)/db = 1
out._backward = _backward
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), "*")
def _backward():
self.grad += other.data * out.grad # d(ab)/da = b
other.grad += self.data * out.grad # d(ab)/db = a
out._backward = _backward
return out
def __pow__(self, n):
out = Value(self.data ** n, (self,), f"**{n}")
def _backward():
self.grad += n * (self.data ** (n - 1)) * out.grad
out._backward = _backward
return out
def exp(self):
out = Value(math.exp(self.data), (self,), "exp")
def _backward():
self.grad += out.data * out.grad # d(e^x)/dx = e^x
out._backward = _backward
return out
def log(self):
x = max(self.data, 1e-30) # guard against log(0)
out = Value(math.log(x), (self,), "log")
def _backward():
self.grad += (1.0 / x) * out.grad
out._backward = _backward
return out
def tanh(self):
out = Value(math.tanh(self.data), (self,), "tanh")
def _backward():
self.grad += (1.0 - out.data ** 2) * out.grad
out._backward = _backward
return out
def backward(self):
"""Compute gradients for every Value that produced this one."""
topo, visited = [], set()
def build(v):
if v not in visited:
visited.add(v)
for child in v._prev:
build(child)
topo.append(v)
build(self)
self.grad = 1.0 # d(output)/d(output) = 1
for v in reversed(topo):
v._backward()
v._backward = lambda: None # free the graph as we go: breaking the
v._prev = () # closure cycles lets Python reclaim memory
# immediately (one backward() per graph)
topo.clear(); visited.clear() # `build` refers to itself, so it would keep the graph alive
# helper operators so `2 * v`, `v - 1` and `v / 2` work
def __radd__(self, other): return self + other
def __rmul__(self, other): return self * other
def __neg__(self): return self * -1
def __sub__(self, other): return self + (-other)
def __truediv__(self, other):
other = other if isinstance(other, Value) else Value(other)
return self * (other ** -1)
Value.__radd__ = __radd__
Value.__rmul__ = __rmul__
Value.__neg__ = __neg__
Value.__sub__ = __sub__
Value.__truediv__ = __truediv__
def dot(xs, ws):
"""Dot product as ONE graph node (instead of 2n tiny + and * nodes).
This single trick makes the model ~100x cheaper in memory and time."""
out = Value(sum(x.data * w.data for x, w in zip(xs, ws)),
tuple(xs) + tuple(ws), "dot")
def _backward():
g = out.grad
for x, w in zip(xs, ws):
x.grad += w.data * g # d(xΒ·w)/dx = w
w.grad += x.data * g # d(xΒ·w)/dw = x
out._backward = _backward
return out
model.py
"""model.py β model definition, primitives, forward pass, checkpoints."""
import math, random, struct
from micrograd import Value, dot
# ---- the shape (small enough to train in pure Python) ----
CTX_LEN = 32 # characters visible at once
D_MODEL = 32 # width of each character's vector
N_LAYERS = 2 # Transformer blocks, stacked
N_HEADS = 2 # attention heads per block
D_FF = 128 # FFN inner width (4x D_MODEL)
HEAD_DIM = D_MODEL // N_HEADS
# ---- initialization ----
def init_matrix(rows, cols, scale=0.02):
return [[Value(random.gauss(0, scale)) for _ in range(cols)] for _ in range(rows)]
def init_model(vocab_size):
p = {}
p["tok_embed"] = init_matrix(vocab_size, D_MODEL, scale=0.1)
p["pos_embed"] = init_matrix(CTX_LEN, D_MODEL, scale=0.1)
for i in range(N_LAYERS):
p[f"ln1_g_{i}"] = [Value(1.0) for _ in range(D_MODEL)]
p[f"ln1_b_{i}"] = [Value(0.0) for _ in range(D_MODEL)]
p[f"W_qkv_{i}"] = init_matrix(D_MODEL, 3 * D_MODEL)
p[f"W_out_{i}"] = init_matrix(D_MODEL, D_MODEL)
p[f"ln2_g_{i}"] = [Value(1.0) for _ in range(D_MODEL)]
p[f"ln2_b_{i}"] = [Value(0.0) for _ in range(D_MODEL)]
p[f"W_ff1_{i}"] = init_matrix(D_MODEL, D_FF)
p[f"W_ff2_{i}"] = init_matrix(D_FF, D_MODEL)
p["ln_f_g"] = [Value(1.0) for _ in range(D_MODEL)]
p["ln_f_b"] = [Value(0.0) for _ in range(D_MODEL)]
p["W_head"] = init_matrix(D_MODEL, vocab_size)
return p
def all_params(params):
for v in params.values():
for row in v:
if isinstance(row, list):
yield from row
else:
yield row
# ---- primitives ----
def matmul(X, W):
"""X: list of rows, W: list of rows. Returns X @ W."""
cols = list(zip(*W)) # columns of W
return [[dot(x_row, col) for col in cols] for x_row in X]
def add_rows(X, Y):
return [[a + b for a, b in zip(rx, ry)] for rx, ry in zip(X, Y)]
def softmax(row):
m = max(v.data for v in row) # subtract max for stability
shifted = [v - m for v in row]
exps = [v.exp() for v in shifted]
total = sum(exps, Value(0.0))
return [v / total for v in exps]
def layernorm(row, gamma, beta, eps=1e-5):
n = len(row)
mean = sum(row, Value(0.0)) / n
diffs = [v - mean for v in row]
var = sum((d * d for d in diffs), Value(0.0)) / n
denom = (var + eps) ** 0.5
norm = [d / denom for d in diffs]
return [g * x + b for g, x, b in zip(gamma, norm, beta)]
def gelu_approx(x):
c = 0.7978845608 # sqrt(2/pi)
return 0.5 * x * (1.0 + ((x + 0.044715 * x * x * x) * c).tanh())
# ---- attention + block ----
def attention(h, W_qkv, W_out):
T = len(h)
qkv = matmul(h, W_qkv) # (T, 3 * D_MODEL)
scale = 1.0 / math.sqrt(HEAD_DIM)
out = [[] for _ in range(T)]
for head in range(N_HEADS):
lo = head * HEAD_DIM
q = [r[lo : lo + HEAD_DIM] for r in qkv]
k = [r[D_MODEL + lo : D_MODEL + lo + HEAD_DIM] for r in qkv]
v = [r[2 * D_MODEL + lo : 2 * D_MODEL + lo + HEAD_DIM] for r in qkv]
for t in range(T):
# causal mask: position t only looks at positions 0..t
scores = [dot(q[t], k[s]) * scale for s in range(t + 1)]
weights = softmax(scores)
for d in range(HEAD_DIM):
out[t].append(dot(weights, [v[s][d] for s in range(t + 1)]))
return matmul(out, W_out) # heads are concatenated in `out`
def block(h, p, i):
# 1. attention sublayer
a = [layernorm(row, p[f"ln1_g_{i}"], p[f"ln1_b_{i}"]) for row in h]
h = add_rows(h, attention(a, p[f"W_qkv_{i}"], p[f"W_out_{i}"]))
# 2. feed-forward sublayer
a = [layernorm(row, p[f"ln2_g_{i}"], p[f"ln2_b_{i}"]) for row in h]
ff = matmul(a, p[f"W_ff1_{i}"]) # (T, D_FF)
ff = [[gelu_approx(x) for x in row] for row in ff]
ff = matmul(ff, p[f"W_ff2_{i}"]) # (T, D_MODEL)
return add_rows(h, ff)
def forward(tokens, p):
h = [[p["tok_embed"][tok][d] + p["pos_embed"][t][d] for d in range(D_MODEL)]
for t, tok in enumerate(tokens)]
for i in range(N_LAYERS):
h = block(h, p, i)
h = [layernorm(row, p["ln_f_g"], p["ln_f_b"]) for row in h]
return matmul(h, p["W_head"]) # (T, vocab)
# ---- checkpoints ----
SHAPE = (CTX_LEN, D_MODEL, N_LAYERS, N_HEADS, D_FF)
def save_model(path, params, chars, step, loss):
with open(path, "wb") as f:
f.write(struct.pack("<5I", *SHAPE)) # so we can detect a mismatch
f.write(struct.pack("<I", len(chars)))
for c in chars:
b = c.encode("utf-8") # 1 to 4 bytes per character
f.write(struct.pack("<B", len(b)) + b) # length prefix, then bytes
f.write(struct.pack("<If", step, loss))
for p in all_params(params):
f.write(struct.pack("<f", p.data))
def load_model(path):
with open(path, "rb") as f:
if struct.unpack("<5I", f.read(20)) != SHAPE:
raise SystemExit("model.bin was trained with different constants")
n_chars = struct.unpack("<I", f.read(4))[0]
chars = []
for _ in range(n_chars):
n = f.read(1)[0]
chars.append(f.read(n).decode("utf-8"))
step, loss = struct.unpack("<If", f.read(8))
params = init_model(n_chars + 1) # +1 for <unk>
for p in all_params(params):
p.data = struct.unpack("<f", f.read(4))[0]
return params, chars, step, loss
train.py
#!/usr/bin/env python3
"""
train.py β train the tiny model in pure Python.
Usage:
python train.py corpus.txt model.bin
"""
import sys, time, random, gc
from pathlib import Path
from micrograd import Value
from model import CTX_LEN, init_model, all_params, forward, softmax, save_model
BATCH_SIZE = 4
STEPS = 300
LR = 3e-3
def compute_loss(logits, targets):
"""Cross-entropy, averaged over positions. logits: (T, vocab), targets: (T,)."""
total = Value(0.0)
for t in range(len(logits)):
probs = softmax(logits[t])
total = total - probs[targets[t]].log()
return total / len(logits)
def main():
random.seed(0)
gc.disable() # the graph is freed by refcount; the cycle collector only slows us down
corpus_path = Path(sys.argv[1] if len(sys.argv) > 1 else "corpus.txt")
out_path = Path(sys.argv[2] if len(sys.argv) > 2 else "model.bin")
text = corpus_path.read_text(encoding="utf-8")
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
data = [stoi[c] for c in text]
if len(data) <= CTX_LEN + 1:
raise SystemExit(f"Corpus too short: {len(data)} chars")
params = init_model(len(chars) + 1) # +1 for <unk>
plist = list(all_params(params))
print(f"Vocab: {len(chars) + 1} Tokens: {len(data):,}")
print(f"Params: {len(plist):,}")
m = [0.0] * len(plist) # Adam: running mean of gradients
v = [0.0] * len(plist) # Adam: running mean of squared gradients
b1, b2, eps = 0.9, 0.999, 1e-8
t0, recent = time.time(), []
print(f"Training for {STEPS} steps...")
for step in range(1, STEPS + 1):
# 1. sample a batch of windows
loss = Value(0.0)
for _ in range(BATCH_SIZE):
j = random.randint(0, len(data) - CTX_LEN - 1)
x = data[j : j + CTX_LEN]
y = data[j + 1 : j + CTX_LEN + 1]
# 2. forward pass
loss = loss + compute_loss(forward(x, params), y)
loss = loss / BATCH_SIZE
# 3. backward pass
loss.backward()
# 4. Adam update
for i, p in enumerate(plist):
m[i] = b1 * m[i] + (1 - b1) * p.grad
v[i] = b2 * v[i] + (1 - b2) * p.grad ** 2
m_hat = m[i] / (1 - b1 ** step)
v_hat = v[i] / (1 - b2 ** step)
p.data -= LR * m_hat / (v_hat ** 0.5 + eps)
p.grad = 0.0
recent.append(loss.data)
if step == 1 or step % 50 == 0:
avg = sum(recent[-50:]) / len(recent[-50:])
print(f"step {step:4d} loss {avg:.4f}")
final = sum(recent[-50:]) / len(recent[-50:])
save_model(out_path, params, chars, STEPS, final)
print(f"Saved: {out_path} (avg loss over last 50 steps: {final:.4f})")
print(f"Done in {time.time() - t0:.1f}s")
if __name__ == "__main__":
main()
infer.py
#!/usr/bin/env python3
"""infer.py β generate text from a trained model."""
import sys, random, gc
from pathlib import Path
from micrograd import Value
from model import CTX_LEN, forward, softmax, load_model
def generate(params, chars, prompt, n_tokens=150, temperature=0.8, top_k=40):
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for i, c in enumerate(chars)}
unk = len(chars)
tokens = [stoi.get(c, unk) for c in prompt] or [unk]
for _ in range(n_tokens):
# 1. forward pass on the last CTX_LEN tokens
gc.disable() # a forward pass allocates a huge graph;
logits = forward(tokens[-CTX_LEN:], params) # collecting mid-way is very slow
nxt = [Value(v.data) for v in logits[-1]] # keep only the numbers we need
del logits
gc.enable(); gc.collect() # nothing calls backward(), so free it here
# 2. temperature scaling
nxt = [v * (1.0 / max(temperature, 1e-6)) for v in nxt]
# 3. top-k filtering: keep only the k largest logits
if top_k > 0:
cutoff = sorted((v.data for v in nxt), reverse=True)[:top_k][-1]
nxt = [v if v.data >= cutoff else Value(-1e9) for v in nxt]
# 4. softmax -> probabilities
probs = softmax(nxt)
# 5. sample one token from the distribution
r, cumsum = random.random(), 0.0
choice = len(probs) - 1
for i, p in enumerate(probs):
cumsum += p.data
if cumsum >= r:
choice = i
break
tokens.append(choice)
return "".join(itos.get(t, "<unk>") for t in tokens)
if __name__ == "__main__":
random.seed(0)
model_path = Path("model.bin")
if not model_path.exists():
raise SystemExit("model.bin not found β run train.py first")
params, chars, step, loss = load_model(model_path)
print(f"Loaded: step={step} loss={loss:.3f}")
if len(sys.argv) > 1:
print(generate(params, chars, sys.argv[1]))
else:
temperature, top_k = 0.8, 40
print("Type a prompt. :t 0.9 sets temperature, :k 20 sets top-k, empty line quits.")
while True:
try:
line = input("> ")
except (EOFError, KeyboardInterrupt):
break
if line.strip() == "" or line.strip() in ("exit", "quit"):
break
if line.startswith(":t "):
temperature = float(line[3:]); continue
if line.startswith(":k "):
top_k = int(line[3:]); continue
print(generate(params, chars, line, temperature=temperature, top_k=top_k))
print()
build_corpus.py
#!/usr/bin/env python3
"""build_corpus.py β concatenate .md/.txt files into corpus.txt."""
import argparse, re
from pathlib import Path
def build_corpus(src, out):
files = sorted(src.rglob("*")) if src.is_dir() else [src]
parts = []
for f in files:
if not f.is_file() or f.suffix.lower() not in {".md", ".txt"}:
continue
text = f.read_text(encoding="utf-8")
text = re.sub(r"[ \t]+", " ", text)
text = re.sub(r"\n{3,}", "\n\n", text).strip()
if text:
parts.append(text)
if not parts:
raise SystemExit(f"No .md/.txt files found under {src}")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text("\n\n".join(parts), encoding="utf-8")
total = sum(len(p) for p in parts)
print(f"Corpus: {out} ({len(parts)} files, {total:,} chars)")
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("src", type=Path)
ap.add_argument("out", type=Path)
args = ap.parse_args()
build_corpus(args.src, args.out)
Want to smoke-test first β set STEPS = 50 (about 3β4 minutes in my container).
Loss stays near 4.65 (ln of the vocab size) β the model isn't learning; check that LR and the corpus are what you think they are.
Output is repetitive β raise temperature in the REPL: :t 0.9 or :t 1.0.
Output looks like word-shaped noise β expected at this size and loss (~2.7). More steps, a bigger model, or a longer context all help, and all cost time.
Out of memory β lower CTX_LEN or BATCH_SIZE.
That's the whole thing
1. Backpropagation is just the chain rule applied in reverse topological order. The autograd engine in Step 1 is the whole algorithm.
2. Transformer language models β from this tiny one to today's largest β are built from the same few primitives: matmul, add, softmax, a normalization layer and an activation. Newer models swap in variants (RMSNorm, SwiGLU, RoPE), but the structure is the same. Stack them, train them, sample from them.
You just built a language model with no libraries, no GPU, and no framework. Every operation is visible. That's the point: once you've seen the mechanism, the frameworks stop being magic.