Start here

Build a tiny AI from scratch

Seven short steps to build a working character-level Transformer using only the Python standard library. No frameworks, no tensor libraries, no dependencies. Every gradient, every matrix multiply, every update is written out β€” so you can see what a neural network actually is.

What it is

A tiny Transformer

2 blocks, 2 heads, 32-wide embeddings, ~33K parameters β€” a scaled-down GPT. Same architecture, small enough to train in pure Python.

What it does

Learns from text

Reads corpus.txt, predicts the next character. Loss drops from ~4.7 β†’ ~2.7 over 300 steps (perplexity β‰ˆ105 β†’ β‰ˆ14). Expect word-shaped fragments, not sentences.

What you get

Five runnable files

micrograd.py, model.py, train.py, infer.py, build_corpus.py. No dependencies beyond Python itself.

given some text β†’ guess the next character every step shown; nothing is hidden behind a library call
Begin ↓

Build a tiny autograd engine

A neural network needs two things: run operations on big collections of numbers, and compute derivatives automatically. We're going to build both from scratch. Step 1 builds the derivative part β€” an autograd engine in ~100 lines.

The core idea: every value is stored in a small object called a Value. It remembers the numbers that produced it, so it can later compute "how much did I contribute to the final loss?" That's what backpropagation is.

micrograd.py β€” the autograd engine
import math


class Value:
    """A number that remembers how it was made."""

    def __init__(self, data, children=(), op=""):
        self.data      = data
        self.grad      = 0.0                 # d(output)/d(this value)
        self._prev     = children            # values that produced this one
        self._op       = op
        self._backward = lambda: None

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad  += out.grad           # d(a+b)/da = 1
            other.grad += out.grad           # d(a+b)/db = 1
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad  += other.data * out.grad   # d(ab)/da = b
            other.grad += self.data  * out.grad   # d(ab)/db = a
        out._backward = _backward
        return out

    def __pow__(self, n):
        out = Value(self.data ** n, (self,), f"**{n}")
        def _backward():
            self.grad += n * (self.data ** (n - 1)) * out.grad
        out._backward = _backward
        return out

    def exp(self):
        out = Value(math.exp(self.data), (self,), "exp")
        def _backward():
            self.grad += out.data * out.grad      # d(e^x)/dx = e^x
        out._backward = _backward
        return out

    def log(self):
        x = max(self.data, 1e-30)                 # guard against log(0)
        out = Value(math.log(x), (self,), "log")
        def _backward():
            self.grad += (1.0 / x) * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        out = Value(math.tanh(self.data), (self,), "tanh")
        def _backward():
            self.grad += (1.0 - out.data ** 2) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        """Compute gradients for every Value that produced this one."""
        topo, visited = [], set()
        def build(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)
        build(self)

        self.grad = 1.0                           # d(output)/d(output) = 1
        for v in reversed(topo):
            v._backward()
            v._backward = lambda: None            # free the graph as we go: breaking the
            v._prev = ()                          # closure cycles lets Python reclaim memory
                                                  # immediately (one backward() per graph)
        topo.clear(); visited.clear()             # `build` refers to itself, so it would keep the graph alive


# helper operators so `2 * v`, `v - 1` and `v / 2` work
def __radd__(self, other): return self + other
def __rmul__(self, other): return self * other
def __neg__(self): return self * -1
def __sub__(self, other): return self + (-other)
def __truediv__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    return self * (other ** -1)
Value.__radd__    = __radd__
Value.__rmul__    = __rmul__
Value.__neg__     = __neg__
Value.__sub__     = __sub__
Value.__truediv__ = __truediv__


def dot(xs, ws):
    """Dot product as ONE graph node (instead of 2n tiny + and * nodes).
    This single trick makes the model ~100x cheaper in memory and time."""
    out = Value(sum(x.data * w.data for x, w in zip(xs, ws)),
                tuple(xs) + tuple(ws), "dot")
    def _backward():
        g = out.grad
        for x, w in zip(xs, ws):
            x.grad += w.data * g                  # d(xΒ·w)/dx = w
            w.grad += x.data * g                  # d(xΒ·w)/dw = x
    out._backward = _backward
    return out

That's the entire autograd engine. Every arithmetic operation returns a new Value that stores two things: the answer (.data), and a small closure that knows how to send the gradient back to the inputs (._backward).

When you call .backward(), the engine walks the computation graph in reverse topological order and applies each ._backward in turn. That's backpropagation. There is nothing else.

The last function, dot, is a speed trick rather than new math: it computes a whole dot product as one graph node with one backward closure, instead of one multiply node and one add node per term. Everything in Step 3 is built on it. Two practical notes: backward() frees the graph as it walks it (so call it once per forward pass, and inspect the graph before calling it), and it clears its own helper state so Python can reclaim the memory immediately.

Test it: a = Value(2); b = Value(3); c = a * b; c.backward() β€” then check a.grad == 3 and b.grad == 2. Chain rule, done.

This is slow. On purpose.

A production framework does the same thing, but with GPU tensors and C-level loops. The math is identical. Here every Value is a Python object (about 500 bytes), so time and memory scale with the number of graph nodes. That is why the lesson shrinks the model to ~33K parameters and fuses each dot product into one node. A naive version that creates a separate node for every multiply and add ran out of memory on a 4 GB machine after just 16 tokens, even for a 64-wide model.

Define the model's shape

Same architecture as a modern language model, just smaller. Every constant is deliberately chosen so the model fits in a few tens of thousands of parameters β€” small enough that pure-Python training finishes in tens of minutes.

model.py β€” constants
CTX_LEN    = 32     # characters visible at once
D_MODEL    = 32     # width of each character's vector
N_LAYERS   = 2      # Transformer blocks, stacked
N_HEADS    = 2      # attention heads per block
D_FF       = 128    # FFN inner width (4x D_MODEL)
HEAD_DIM   = D_MODEL // N_HEADS   # 16 dims per head

# ---- how training runs ----
BATCH_SIZE = 4      # windows per step
STEPS      = 300    # total update steps
LR         = 3e-3   # Adam learning rate

The vocab depends on the corpus. For a typical 1M-character text, vocab is ~90–120 characters (plus <unk>). For the reference corpus in this lesson (467K characters of Python’s built-in documentation text), vocab = 105 including <unk>. That gives 32,640 parameters, about 33K. Run train.py and it prints the exact count. (The embedding table and the output head each add D_MODEL parameters per vocabulary entry.)

The math of one forward pass

Every symbol is one line in the code. Shapes are for one window with T = 64 tokens.

h = tok_embed(x) + pos_embed(position) (T, D_MODEL)
for each block:
  a = LayerNorm(h) (T, D_MODEL)
  q, k, v = split(a @ W_qkv) (T, D_MODEL) each
  A = softmax(qΒ·kα΅€ / √HEAD_DIM, causal), per head (T, T)
  h = h + concat(A @ v over heads) @ W_out (T, D_MODEL)
  h = h + FFN(LayerNorm(h)) (T, D_MODEL)
logits = LayerNorm(h) @ W_head (T, vocab)

Every @ is a matrix multiply built from dot products in Step 3. Nothing hidden.

Same architecture, smaller numbers

The only difference from a full-size model is size: D_MODEL=32 instead of 768, N_LAYERS=2 instead of 12, CTX_LEN=32 instead of 1024. The core math is the same (GPT-2 also adds biases to its linear layers and ties its input and output embeddings). Everything you learn here transfers directly to the full-size version.

Forward pass β€” every operation

A Transformer’s forward pass is built from five primitives (plus embedding lookups): matmul, add, softmax, LayerNorm, and activation. Each one is a short function below, using only lists of Value objects.

model.py β€” primitives
from micrograd import Value, dot
import math, random

# ---- primitives ----
def matmul(X, W):
    """X: list of rows, W: list of rows. Returns X @ W."""
    cols = list(zip(*W))                          # columns of W
    return [[dot(x_row, col) for col in cols] for x_row in X]


def add_rows(X, Y):
    return [[a + b for a, b in zip(rx, ry)] for rx, ry in zip(X, Y)]


def softmax(row):
    m       = max(v.data for v in row)            # subtract max for stability
    shifted = [v - m for v in row]
    exps    = [v.exp() for v in shifted]
    total   = sum(exps, Value(0.0))
    return [v / total for v in exps]


def layernorm(row, gamma, beta, eps=1e-5):
    n     = len(row)
    mean  = sum(row, Value(0.0)) / n
    diffs = [v - mean for v in row]
    var   = sum((d * d for d in diffs), Value(0.0)) / n
    denom = (var + eps) ** 0.5
    norm  = [d / denom for d in diffs]
    return [g * x + b for g, x, b in zip(gamma, norm, beta)]


def gelu_approx(x):
    c = 0.7978845608                              # sqrt(2/pi)
    return 0.5 * x * (1.0 + ((x + 0.044715 * x * x * x) * c).tanh())

Why these five? Nearly every operation here decomposes into them. A QKV projection is a single matmul (this model has no biases). An attention score is a matmul plus a divide plus a softmax. LayerNorm is a mean plus a variance plus a rescale. The FFN is two matmuls separated by a GELU.

Why matmul uses dot: each output element is one dot product of a row of X with a column of W, so it becomes a single graph node. That keeps the graph small enough to fit in memory and is the reason this model trains at all in pure Python.

Why lists of Value? Because Value remembers how it was made, so calling .backward() on the final loss automatically flows gradients back through every primitive β€” including through every dot product. You write the forward pass; the gradients are free.

The Transformer block

Now compose the primitives into a block. Two sublayers, each with a residual connection.

model.py β€” the block
# ---- attention + block ----
def attention(h, W_qkv, W_out):
    T   = len(h)
    qkv = matmul(h, W_qkv)                        # (T, 3 * D_MODEL)
    scale = 1.0 / math.sqrt(HEAD_DIM)
    out = [[] for _ in range(T)]

    for head in range(N_HEADS):
        lo = head * HEAD_DIM
        q = [r[lo : lo + HEAD_DIM]                          for r in qkv]
        k = [r[D_MODEL + lo : D_MODEL + lo + HEAD_DIM]      for r in qkv]
        v = [r[2 * D_MODEL + lo : 2 * D_MODEL + lo + HEAD_DIM] for r in qkv]

        for t in range(T):
            # causal mask: position t only looks at positions 0..t
            scores  = [dot(q[t], k[s]) * scale for s in range(t + 1)]
            weights = softmax(scores)
            for d in range(HEAD_DIM):
                out[t].append(dot(weights, [v[s][d] for s in range(t + 1)]))

    return matmul(out, W_out)                     # heads are concatenated in `out`


def block(h, p, i):
    # 1. attention sublayer
    a = [layernorm(row, p[f"ln1_g_{i}"], p[f"ln1_b_{i}"]) for row in h]
    h = add_rows(h, attention(a, p[f"W_qkv_{i}"], p[f"W_out_{i}"]))

    # 2. feed-forward sublayer
    a  = [layernorm(row, p[f"ln2_g_{i}"], p[f"ln2_b_{i}"]) for row in h]
    ff = matmul(a, p[f"W_ff1_{i}"])               # (T, D_FF)
    ff = [[gelu_approx(x) for x in row] for row in ff]
    ff = matmul(ff, p[f"W_ff2_{i}"])              # (T, D_MODEL)
    return add_rows(h, ff)


def forward(tokens, p):
    h = [[p["tok_embed"][tok][d] + p["pos_embed"][t][d] for d in range(D_MODEL)]
         for t, tok in enumerate(tokens)]
    for i in range(N_LAYERS):
        h = block(h, p, i)
    h = [layernorm(row, p["ln_f_g"], p["ln_f_b"]) for row in h]
    return matmul(h, p["W_head"])                 # (T, vocab)

The full forward pass is: embed β†’ block β†’ block β†’ final LayerNorm β†’ linear projection to vocab. That's four lines of code, given the functions above.

h = tok_embed(x) + pos_embed(positions)
h = block(h, p, 0)
h = block(h, p, 1)
h = [layernorm(row, p["ln_f_g"], p["ln_f_b"]) for row in h]
logits = matmul(h, p["W_head"]) (T, vocab)
How the heads work

With N_HEADS=2 and D_MODEL=32, each head works on its own 16-dim slice of q, k and v, with its own attention weights. The two heads’ outputs are concatenated and mixed by W_out. The causal mask is implemented by only scoring positions 0..t for position t, so future characters never enter the softmax.

Backward pass β€” free gradients

Here's the beautiful part: you don't write the backward pass. The autograd engine from Step 1 does it. You just call .backward() on the loss.

train.py β€” loss and backward
def compute_loss(logits, targets):
    """Cross-entropy loss. logits: (T, vocab), targets: (T,)."""
    total = Value(0.0)
    for t in range(len(logits)):
        probs = softmax(logits[t])
        total = total - probs[targets[t]].log()
    return total / len(logits)


# one full training step (plain gradient descent for now; Step 5 upgrades it to Adam)
logits = forward(x, params)  # (T, vocab) list of Values
loss   = compute_loss(logits, y)
loss.backward()              # <- everything happens here

for p in all_params(params):
    p.data -= LR * p.grad    # nudge each weight downhill
    p.grad  = 0.0            # reset for next step

That's the entire backward pass. When you call loss.backward(), the autograd engine walks the computation graph from Step 3, and at every Value, executes the small closure that was registered when the value was created. The gradients flow backward through matmuls, through softmax, through LayerNorm, through GELU β€” all automatically.

The only place you need to be careful is that p.grad accumulates across backward calls. That's why you reset to 0.0 after each update. Also remember that backward() frees the graph as it goes, so call it once per forward pass.

Why this works

Every operation in the forward pass is a Value that knows how its output changes when each input changes. Backprop is just the chain rule applied in reverse topological order. The autograd engine collects the results. No manual gradient derivation required.

Train β€” the loop

Every training step is the same four operations: sample, forward, backward, update. The update rule is written out explicitly.

train.py β€” the loop
# Adam optimizer state (one entry per parameter)
plist = list(all_params(params))
m = [0.0] * len(plist)       # running mean of gradients
v = [0.0] * len(plist)       # running mean of squared gradients
b1, b2, eps = 0.9, 0.999, 1e-8

for step in range(1, STEPS + 1):
    # 1. sample a batch of windows, 2. forward pass on each
    loss = Value(0.0)
    for _ in range(BATCH_SIZE):
        j = random.randint(0, len(data) - CTX_LEN - 1)
        x = data[j : j + CTX_LEN]
        y = data[j + 1 : j + CTX_LEN + 1]
        loss = loss + compute_loss(forward(x, params), y)
    loss = loss / BATCH_SIZE

    # 3. backward pass - the whole point of Step 1
    loss.backward()

    # 4. Adam update (written out explicitly)
    for i, p in enumerate(plist):
        m[i] = b1 * m[i] + (1 - b1) * p.grad
        v[i] = b2 * v[i] + (1 - b2) * p.grad ** 2
        m_hat = m[i] / (1 - b1 ** step)
        v_hat = v[i] / (1 - b2 ** step)
        p.data -= LR * m_hat / (v_hat ** 0.5 + eps)
        p.grad  = 0.0

    if step == 1 or step % 50 == 0:
        print(f"step {step:4d}  loss {loss.data:.4f}")

What you can see that a framework hides:

β‘  The optimizer state. Adam keeps two running averages per parameter (m and v). A framework hides these inside a built-in optimizer object; here they're plain lists.

β‘‘ The bias correction. m_hat and v_hat are divided by 1 - b^step to correct for the fact that at step 1 the running averages are still close to zero. Without it, the first few updates would be far too small.

β‘’ The reset. p.grad = 0.0 after each update. Forget this and gradients accumulate across steps, so each update uses a pile of stale gradients and training goes wrong.

β‘£ The garbage collector. train.py calls gc.disable(). Each step allocates millions of small objects that reference each other; Python’s cycle collector kept rescanning them, and in my test the forward pass was about 10Γ— slower with it on. Because backward() frees the graph as it goes, memory stays flat without it.

What you'll see

Loss trajectory from a real run on 467K characters of Python’s built-in documentation text (step 250 omitted; each line after the first is the average of the previous 50 steps):

output from train.py
Vocab: 105   Tokens: 467,243
Params: 32,640
Training for 300 steps...
step    1  loss 4.6803
step   50  loss 3.6219
step  100  loss 3.0556
step  150  loss 2.8065
step  200  loss 2.7181
step  300  loss 2.6631
Saved: model.bin  (avg loss over last 50 steps: 2.6631)
Done in 1143.2s

Loss falls from about 4.7 (random guessing is ln 105 β‰ˆ 4.65) to about 2.7: perplexity drops from β‰ˆ105 to β‰ˆ14. The run only sees ~38K of the corpus’s 467K characters (300 steps Γ— 4 windows Γ— 32), so nearly every batch is new text and training loss is a fair stand-in for held-out loss. It took about 19 minutes in my test container; yours will differ, and the script prints the time. The curve is still falling at step 300, so more steps help β€” this run is short on purpose.

Save and load

With no framework, there's no built-in save/load. We write a simple binary format: a small header, then all the weights as 32-bit floats. That's it β€” that's all a checkpoint is.

train.py / infer.py β€” save and load
import struct

SHAPE = (CTX_LEN, D_MODEL, N_LAYERS, N_HEADS, D_FF)

def save_model(path, params, chars, step, loss):
    with open(path, "wb") as f:
        f.write(struct.pack("<5I", *SHAPE))               # so we can detect a mismatch
        f.write(struct.pack("<I", len(chars)))
        for c in chars:
            b = c.encode("utf-8")                         # 1 to 4 bytes per character
            f.write(struct.pack("<B", len(b)) + b)        # length prefix, then bytes
        f.write(struct.pack("<If", step, loss))
        for p in all_params(params):
            f.write(struct.pack("<f", p.data))


def load_model(path):
    with open(path, "rb") as f:
        if struct.unpack("<5I", f.read(20)) != SHAPE:
            raise SystemExit("model.bin was trained with different constants")
        n_chars = struct.unpack("<I", f.read(4))[0]
        chars = []
        for _ in range(n_chars):
            n = f.read(1)[0]
            chars.append(f.read(n).decode("utf-8"))
        step, loss = struct.unpack("<If", f.read(8))
        params = init_model(n_chars + 1)                  # +1 for <unk>
        for p in all_params(params):
            p.data = struct.unpack("<f", f.read(4))[0]
    return params, chars, step, loss

What's happening: the file starts with the five model constants (so loading a checkpoint with the wrong shape fails loudly), then the number of vocabulary entries, then each character as a length byte followed by its UTF-8 bytes, then the training step and loss, then every parameter as a 4-byte float. Characters are length-prefixed because many (Γ©, β€”, ’) take several bytes in UTF-8; writing one byte per character would corrupt the vocabulary.

Why this format? Because it's the smallest thing that could work. More complex formats exist for good reasons (arbitrary tensor shapes, dtypes, device placement, metadata), but none of that matters here. The 32,640 parameters take exactly 130,560 bytes; the reference model.bin is 130,816 bytes once the header and vocabulary are included. Weights are stored as 32-bit floats, so they are rounded slightly on save.

The structure of a checkpoint, laid bare

Every modern model file (safetensors, GGUF, and others) is essentially this: a header describing shapes and dtypes, followed by the raw weights. The format matters for interoperability, not for correctness. Once you've written this short saver, you understand what every checkpoint loader actually does.

Generate text

Load the weights, feed a prompt, predict the next character, append it, repeat. Every step is a forward pass β€” no backward, no gradients.

infer.py β€” generate
def generate(params, chars, prompt, n_tokens=150, temperature=0.8, top_k=40):
    stoi = {c: i for i, c in enumerate(chars)}
    itos = {i: c for i, c in enumerate(chars)}
    unk  = len(chars)

    tokens = [stoi.get(c, unk) for c in prompt] or [unk]

    for _ in range(n_tokens):
        # 1. forward pass on the last CTX_LEN tokens
        gc.disable()                                          # a forward pass allocates a huge graph;
        logits = forward(tokens[-CTX_LEN:], params)           # collecting mid-way is very slow
        nxt    = [Value(v.data) for v in logits[-1]]          # keep only the numbers we need
        del logits
        gc.enable(); gc.collect()                             # nothing calls backward(), so free it here

        # 2. temperature scaling
        nxt = [v * (1.0 / max(temperature, 1e-6)) for v in nxt]

        # 3. top-k filtering: keep only the k largest logits
        if top_k > 0:
            cutoff = sorted((v.data for v in nxt), reverse=True)[:top_k][-1]
            nxt = [v if v.data >= cutoff else Value(-1e9) for v in nxt]

        # 4. softmax -> probabilities
        probs = softmax(nxt)

        # 5. sample one token from the distribution
        r, cumsum = random.random(), 0.0
        choice = len(probs) - 1
        for i, p in enumerate(probs):
            cumsum += p.data
            if cumsum >= r:
                choice = i
                break
        tokens.append(choice)

    return "".join(itos.get(t, "<unk>") for t in tokens)

A note on speed: generation also builds a graph, so it turns the garbage collector off during the forward pass and collects once per character. That took generation from about 2.5 s to about 0.34 s per character in my container, so 150 characters take about a minute.

Step 5 is the interesting one. A one-line sampler call does the same thing in a few characters. Here you can see the algorithm: draw a uniform random number between 0 and 1, then walk the probability distribution accumulating until you pass it. That's called inverse transform sampling, and it's what every sampler uses.

Why the loop instead of random.choices(population, weights)? Because random.choices does the same thing internally, and seeing the loop makes it clear that "sampling from a distribution" isn't magic β€” it's a linear scan.

A real session with the trained model. :t sets temperature and :k sets top-k, and the model is only 32,640 parameters at loss β‰ˆ2.7, so expect English-shaped fragments rather than words:

infer.py β€” sample session
> :t 0.5
> :k 10
> The
The t atharexthe te are is lle s the e tiontes on ontre ase ss on thed te the ctice as le ioftins at astirertins as. sthertere the an ons ante iconcr inin
Why 32 tokens is short

The model sees at most 32 characters β€” about 6 words. It can't complete a sentence, let alone follow a paragraph. This is the biggest limitation of the model, and it comes from CTX_LEN = 32 combined with character-level tokens. A longer context costs time: attention work grows with the square of the window.

That's the whole build

  1. Autograd β€” ~100 lines. Enables everything else.
  2. Shape β€” small enough to run in pure Python (~33K params).
  3. Forward β€” 5 primitives composed into a Transformer.
  4. Backward β€” free, courtesy of the autograd engine.
  5. Train β€” Adam written out explicitly.
  6. Save β€” 20-line binary format.
  7. Generate β€” sample loop, no gradient.

What is this model actually good for?

It's not a product. It's a microscope. Here's what you can genuinely do with it:

  • See what autograd really is. Print any intermediate Value, inspect its .grad after backward. This is impossible in a framework β€” you'd have to add hooks.
  • Watch gradients flow through attention. After loss.backward(), inspect params["W_qkv_0"][0][0].grad (before the update loop zeroes it) and see how it responds to the loss. That chain is the entire learning mechanism.
  • Study the optimizer. Raise LR from 3e-3 to 3e-2 and compare the loss curve, then lower it to 3e-4 and compare again. The relationship is direct because you can read the update rule.
  • Understand sampling. Try :t 0.1 (almost always the single most likely character, close to greedy decoding) against :t 1.5 (much more random).
  • Understand checkpoints. The saver is short. Add a feature (a new field, a compression) and understand exactly what happens to the file.

What it can't do: anything that requires real scale. This is a teaching model, not a shipping model. But the core ideas carry over to modern language models; the biggest differences are scale and a handful of refinements (see Stage 4).

Where to get a corpus

Project Gutenberg has thousands of free public-domain books. Wikipedia has dumps. Or any folder of your own .md or .txt files. Aim for at least a few hundred thousand characters (the reference run used 467K). The reference corpus is Python's own documentation text: python -c "from pydoc_data.topics import topics; open('docs/py.md','w',encoding='utf-8').write('\n\n'.join(topics.values()))"

How to scale it up

The architecture you just built is GPT-2's architecture, minus biases and tied embeddings, scaled down by ~3,800Γ— in parameter count. Scaling up is a sequence of one-change-at-a-time experiments. Do them in order.

Stage 1 β€” bigger model, same code

The pure-Python version is ~33K parameters. Doubling D_MODEL to 64 (and D_FF to 256) gives ~114K parameters and roughly 4Γ— the compute per token, so a run that took ~19 minutes would take over an hour (an estimate; I didn't time that configuration). Try it once; you'll appreciate why frameworks exist.

Stage 2 β€” move to a tensor library

Take the same architecture and rewrite it with a tensor library. Now you can:

  • Use tensors instead of lists (often orders of magnitude faster)
  • Call .backward() on tensors instead of Value (identical idea, faster)
  • Use a built-in multi-head attention layer instead of writing it out

What you learn: which parts of a framework are doing real work vs. which parts are convenience. Both are valuable to know.

Stage 3 β€” reproduce GPT-2 small

Same architecture as your tiny model, but at 124M parameters (~3,800Γ— yours). Same code structure, different constants:

D_MODEL=768 Β· N_LAYERS=12 Β· N_HEADS=12 Β· D_FF=3072 Β· CTX_LEN=1024 Β· vocab=50,257
β‰ˆ 124M parameters Β· needs a GPU Β· ~40GB of text Β· BPE tokenizer

At this size you also need a proper tokenizer (subword, not character-level) and a learning-rate schedule. Those are the two biggest new ideas at this stage.

Stage 4 β€” modern block improvements

Llama and Mistral keep the same overall shape but swap four pieces inside each block:

Old (GPT-2)New (Llama)Why
LayerNormRMSNormCheaper, no mean subtraction
Learned position tableRotary positions (RoPE)Extrapolates to longer contexts
GELUSwiGLUBetter loss per parameter
Full multi-head attentionGrouped-query attention (GQA)Smaller KV cache at inference

Stage 5 β€” from base model to assistant

The model you built continues text. To make it answer questions:

  • Supervised fine-tuning β€” train on (question, answer) pairs
  • Preference training (RLHF or DPO) β€” train on rankings
  • Tool use β€” emit structured function calls

None of these change the architecture. They're new training objectives on top of the same base model.

Concrete first moves

  1. Run the five files as-is. Confirm 32,640 parameters and a final loss near 2.7 (it will vary with your corpus).
  2. Print the gradient of a specific weight after one training step. Understand where the number came from.
  3. Double D_MODEL to 64 and D_FF to 256. Retrain. Expect about 3.5Γ— the parameters (~114K) and roughly 4Γ— the training time.
  4. Only then: rewrite with a tensor library and compare speed.

The full source

To run it

Requires: Python 3.9+. No pip install needed. No dependencies beyond the standard library.

Layout: all five files in one folder, plus a corpus.txt.

mkdir docs && cp some_text.md docs/
python build_corpus.py ./docs corpus.txt
python train.py corpus.txt model.bin
python infer.py

Expected: 32,640 parameters with the reference corpus; training took about 19 minutes in my test container (your machine will differ), then ~0.3 s per generated character. Peak memory was about 600 MB. Tested on Python 3.12.

micrograd.py

micrograd.py
"""
micrograd.py β€” a tiny autograd engine.

Every operation returns a Value that knows how to push its gradient
back to the Values that produced it. Call .backward() on any Value.

Usage:
    a = Value(2.0)
    b = Value(3.0)
    c = a * b + a
    c.backward()
    print(a.grad)  # 4.0  (dc/da = b + 1)
    print(b.grad)  # 2.0  (dc/db = a)
"""
import math


class Value:
    """A number that remembers how it was made."""

    def __init__(self, data, children=(), op=""):
        self.data      = data
        self.grad      = 0.0                 # d(output)/d(this value)
        self._prev     = children            # values that produced this one
        self._op       = op
        self._backward = lambda: None

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")
        def _backward():
            self.grad  += out.grad           # d(a+b)/da = 1
            other.grad += out.grad           # d(a+b)/db = 1
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")
        def _backward():
            self.grad  += other.data * out.grad   # d(ab)/da = b
            other.grad += self.data  * out.grad   # d(ab)/db = a
        out._backward = _backward
        return out

    def __pow__(self, n):
        out = Value(self.data ** n, (self,), f"**{n}")
        def _backward():
            self.grad += n * (self.data ** (n - 1)) * out.grad
        out._backward = _backward
        return out

    def exp(self):
        out = Value(math.exp(self.data), (self,), "exp")
        def _backward():
            self.grad += out.data * out.grad      # d(e^x)/dx = e^x
        out._backward = _backward
        return out

    def log(self):
        x = max(self.data, 1e-30)                 # guard against log(0)
        out = Value(math.log(x), (self,), "log")
        def _backward():
            self.grad += (1.0 / x) * out.grad
        out._backward = _backward
        return out

    def tanh(self):
        out = Value(math.tanh(self.data), (self,), "tanh")
        def _backward():
            self.grad += (1.0 - out.data ** 2) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        """Compute gradients for every Value that produced this one."""
        topo, visited = [], set()
        def build(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)
        build(self)

        self.grad = 1.0                           # d(output)/d(output) = 1
        for v in reversed(topo):
            v._backward()
            v._backward = lambda: None            # free the graph as we go: breaking the
            v._prev = ()                          # closure cycles lets Python reclaim memory
                                                  # immediately (one backward() per graph)
        topo.clear(); visited.clear()             # `build` refers to itself, so it would keep the graph alive


# helper operators so `2 * v`, `v - 1` and `v / 2` work
def __radd__(self, other): return self + other
def __rmul__(self, other): return self * other
def __neg__(self): return self * -1
def __sub__(self, other): return self + (-other)
def __truediv__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    return self * (other ** -1)
Value.__radd__    = __radd__
Value.__rmul__    = __rmul__
Value.__neg__     = __neg__
Value.__sub__     = __sub__
Value.__truediv__ = __truediv__


def dot(xs, ws):
    """Dot product as ONE graph node (instead of 2n tiny + and * nodes).
    This single trick makes the model ~100x cheaper in memory and time."""
    out = Value(sum(x.data * w.data for x, w in zip(xs, ws)),
                tuple(xs) + tuple(ws), "dot")
    def _backward():
        g = out.grad
        for x, w in zip(xs, ws):
            x.grad += w.data * g                  # d(xΒ·w)/dx = w
            w.grad += x.data * g                  # d(xΒ·w)/dw = x
    out._backward = _backward
    return out

model.py

model.py
"""model.py β€” model definition, primitives, forward pass, checkpoints."""
import math, random, struct
from micrograd import Value, dot

# ---- the shape (small enough to train in pure Python) ----
CTX_LEN    = 32     # characters visible at once
D_MODEL    = 32     # width of each character's vector
N_LAYERS   = 2      # Transformer blocks, stacked
N_HEADS    = 2      # attention heads per block
D_FF       = 128    # FFN inner width (4x D_MODEL)
HEAD_DIM   = D_MODEL // N_HEADS


# ---- initialization ----
def init_matrix(rows, cols, scale=0.02):
    return [[Value(random.gauss(0, scale)) for _ in range(cols)] for _ in range(rows)]


def init_model(vocab_size):
    p = {}
    p["tok_embed"] = init_matrix(vocab_size, D_MODEL, scale=0.1)
    p["pos_embed"] = init_matrix(CTX_LEN, D_MODEL, scale=0.1)

    for i in range(N_LAYERS):
        p[f"ln1_g_{i}"] = [Value(1.0) for _ in range(D_MODEL)]
        p[f"ln1_b_{i}"] = [Value(0.0) for _ in range(D_MODEL)]
        p[f"W_qkv_{i}"] = init_matrix(D_MODEL, 3 * D_MODEL)
        p[f"W_out_{i}"] = init_matrix(D_MODEL, D_MODEL)
        p[f"ln2_g_{i}"] = [Value(1.0) for _ in range(D_MODEL)]
        p[f"ln2_b_{i}"] = [Value(0.0) for _ in range(D_MODEL)]
        p[f"W_ff1_{i}"] = init_matrix(D_MODEL, D_FF)
        p[f"W_ff2_{i}"] = init_matrix(D_FF, D_MODEL)

    p["ln_f_g"] = [Value(1.0) for _ in range(D_MODEL)]
    p["ln_f_b"] = [Value(0.0) for _ in range(D_MODEL)]
    p["W_head"] = init_matrix(D_MODEL, vocab_size)
    return p


def all_params(params):
    for v in params.values():
        for row in v:
            if isinstance(row, list):
                yield from row
            else:
                yield row


# ---- primitives ----
def matmul(X, W):
    """X: list of rows, W: list of rows. Returns X @ W."""
    cols = list(zip(*W))                          # columns of W
    return [[dot(x_row, col) for col in cols] for x_row in X]


def add_rows(X, Y):
    return [[a + b for a, b in zip(rx, ry)] for rx, ry in zip(X, Y)]


def softmax(row):
    m       = max(v.data for v in row)            # subtract max for stability
    shifted = [v - m for v in row]
    exps    = [v.exp() for v in shifted]
    total   = sum(exps, Value(0.0))
    return [v / total for v in exps]


def layernorm(row, gamma, beta, eps=1e-5):
    n     = len(row)
    mean  = sum(row, Value(0.0)) / n
    diffs = [v - mean for v in row]
    var   = sum((d * d for d in diffs), Value(0.0)) / n
    denom = (var + eps) ** 0.5
    norm  = [d / denom for d in diffs]
    return [g * x + b for g, x, b in zip(gamma, norm, beta)]


def gelu_approx(x):
    c = 0.7978845608                              # sqrt(2/pi)
    return 0.5 * x * (1.0 + ((x + 0.044715 * x * x * x) * c).tanh())


# ---- attention + block ----
def attention(h, W_qkv, W_out):
    T   = len(h)
    qkv = matmul(h, W_qkv)                        # (T, 3 * D_MODEL)
    scale = 1.0 / math.sqrt(HEAD_DIM)
    out = [[] for _ in range(T)]

    for head in range(N_HEADS):
        lo = head * HEAD_DIM
        q = [r[lo : lo + HEAD_DIM]                          for r in qkv]
        k = [r[D_MODEL + lo : D_MODEL + lo + HEAD_DIM]      for r in qkv]
        v = [r[2 * D_MODEL + lo : 2 * D_MODEL + lo + HEAD_DIM] for r in qkv]

        for t in range(T):
            # causal mask: position t only looks at positions 0..t
            scores  = [dot(q[t], k[s]) * scale for s in range(t + 1)]
            weights = softmax(scores)
            for d in range(HEAD_DIM):
                out[t].append(dot(weights, [v[s][d] for s in range(t + 1)]))

    return matmul(out, W_out)                     # heads are concatenated in `out`


def block(h, p, i):
    # 1. attention sublayer
    a = [layernorm(row, p[f"ln1_g_{i}"], p[f"ln1_b_{i}"]) for row in h]
    h = add_rows(h, attention(a, p[f"W_qkv_{i}"], p[f"W_out_{i}"]))

    # 2. feed-forward sublayer
    a  = [layernorm(row, p[f"ln2_g_{i}"], p[f"ln2_b_{i}"]) for row in h]
    ff = matmul(a, p[f"W_ff1_{i}"])               # (T, D_FF)
    ff = [[gelu_approx(x) for x in row] for row in ff]
    ff = matmul(ff, p[f"W_ff2_{i}"])              # (T, D_MODEL)
    return add_rows(h, ff)


def forward(tokens, p):
    h = [[p["tok_embed"][tok][d] + p["pos_embed"][t][d] for d in range(D_MODEL)]
         for t, tok in enumerate(tokens)]
    for i in range(N_LAYERS):
        h = block(h, p, i)
    h = [layernorm(row, p["ln_f_g"], p["ln_f_b"]) for row in h]
    return matmul(h, p["W_head"])                 # (T, vocab)


# ---- checkpoints ----
SHAPE = (CTX_LEN, D_MODEL, N_LAYERS, N_HEADS, D_FF)

def save_model(path, params, chars, step, loss):
    with open(path, "wb") as f:
        f.write(struct.pack("<5I", *SHAPE))               # so we can detect a mismatch
        f.write(struct.pack("<I", len(chars)))
        for c in chars:
            b = c.encode("utf-8")                         # 1 to 4 bytes per character
            f.write(struct.pack("<B", len(b)) + b)        # length prefix, then bytes
        f.write(struct.pack("<If", step, loss))
        for p in all_params(params):
            f.write(struct.pack("<f", p.data))


def load_model(path):
    with open(path, "rb") as f:
        if struct.unpack("<5I", f.read(20)) != SHAPE:
            raise SystemExit("model.bin was trained with different constants")
        n_chars = struct.unpack("<I", f.read(4))[0]
        chars = []
        for _ in range(n_chars):
            n = f.read(1)[0]
            chars.append(f.read(n).decode("utf-8"))
        step, loss = struct.unpack("<If", f.read(8))
        params = init_model(n_chars + 1)                  # +1 for <unk>
        for p in all_params(params):
            p.data = struct.unpack("<f", f.read(4))[0]
    return params, chars, step, loss

train.py

train.py
#!/usr/bin/env python3
"""
train.py β€” train the tiny model in pure Python.

Usage:
    python train.py corpus.txt model.bin
"""
import sys, time, random, gc
from pathlib import Path
from micrograd import Value
from model import CTX_LEN, init_model, all_params, forward, softmax, save_model

BATCH_SIZE = 4
STEPS      = 300
LR         = 3e-3


def compute_loss(logits, targets):
    """Cross-entropy, averaged over positions. logits: (T, vocab), targets: (T,)."""
    total = Value(0.0)
    for t in range(len(logits)):
        probs = softmax(logits[t])
        total = total - probs[targets[t]].log()
    return total / len(logits)


def main():
    random.seed(0)
    gc.disable()          # the graph is freed by refcount; the cycle collector only slows us down
    corpus_path = Path(sys.argv[1] if len(sys.argv) > 1 else "corpus.txt")
    out_path    = Path(sys.argv[2] if len(sys.argv) > 2 else "model.bin")

    text  = corpus_path.read_text(encoding="utf-8")
    chars = sorted(set(text))
    stoi  = {c: i for i, c in enumerate(chars)}
    data  = [stoi[c] for c in text]
    if len(data) <= CTX_LEN + 1:
        raise SystemExit(f"Corpus too short: {len(data)} chars")

    params = init_model(len(chars) + 1)               # +1 for <unk>
    plist  = list(all_params(params))
    print(f"Vocab: {len(chars) + 1}   Tokens: {len(data):,}")
    print(f"Params: {len(plist):,}")

    m = [0.0] * len(plist)                            # Adam: running mean of gradients
    v = [0.0] * len(plist)                            # Adam: running mean of squared gradients
    b1, b2, eps = 0.9, 0.999, 1e-8

    t0, recent = time.time(), []
    print(f"Training for {STEPS} steps...")
    for step in range(1, STEPS + 1):
        # 1. sample a batch of windows
        loss = Value(0.0)
        for _ in range(BATCH_SIZE):
            j = random.randint(0, len(data) - CTX_LEN - 1)
            x = data[j : j + CTX_LEN]
            y = data[j + 1 : j + CTX_LEN + 1]
            # 2. forward pass
            loss = loss + compute_loss(forward(x, params), y)
        loss = loss / BATCH_SIZE

        # 3. backward pass
        loss.backward()

        # 4. Adam update
        for i, p in enumerate(plist):
            m[i] = b1 * m[i] + (1 - b1) * p.grad
            v[i] = b2 * v[i] + (1 - b2) * p.grad ** 2
            m_hat = m[i] / (1 - b1 ** step)
            v_hat = v[i] / (1 - b2 ** step)
            p.data -= LR * m_hat / (v_hat ** 0.5 + eps)
            p.grad  = 0.0

        recent.append(loss.data)
        if step == 1 or step % 50 == 0:
            avg = sum(recent[-50:]) / len(recent[-50:])
            print(f"step {step:4d}  loss {avg:.4f}")

    final = sum(recent[-50:]) / len(recent[-50:])
    save_model(out_path, params, chars, STEPS, final)
    print(f"Saved: {out_path}  (avg loss over last 50 steps: {final:.4f})")
    print(f"Done in {time.time() - t0:.1f}s")


if __name__ == "__main__":
    main()

infer.py

infer.py
#!/usr/bin/env python3
"""infer.py β€” generate text from a trained model."""
import sys, random, gc
from pathlib import Path
from micrograd import Value
from model import CTX_LEN, forward, softmax, load_model


def generate(params, chars, prompt, n_tokens=150, temperature=0.8, top_k=40):
    stoi = {c: i for i, c in enumerate(chars)}
    itos = {i: c for i, c in enumerate(chars)}
    unk  = len(chars)

    tokens = [stoi.get(c, unk) for c in prompt] or [unk]

    for _ in range(n_tokens):
        # 1. forward pass on the last CTX_LEN tokens
        gc.disable()                                          # a forward pass allocates a huge graph;
        logits = forward(tokens[-CTX_LEN:], params)           # collecting mid-way is very slow
        nxt    = [Value(v.data) for v in logits[-1]]          # keep only the numbers we need
        del logits
        gc.enable(); gc.collect()                             # nothing calls backward(), so free it here

        # 2. temperature scaling
        nxt = [v * (1.0 / max(temperature, 1e-6)) for v in nxt]

        # 3. top-k filtering: keep only the k largest logits
        if top_k > 0:
            cutoff = sorted((v.data for v in nxt), reverse=True)[:top_k][-1]
            nxt = [v if v.data >= cutoff else Value(-1e9) for v in nxt]

        # 4. softmax -> probabilities
        probs = softmax(nxt)

        # 5. sample one token from the distribution
        r, cumsum = random.random(), 0.0
        choice = len(probs) - 1
        for i, p in enumerate(probs):
            cumsum += p.data
            if cumsum >= r:
                choice = i
                break
        tokens.append(choice)

    return "".join(itos.get(t, "<unk>") for t in tokens)


if __name__ == "__main__":
    random.seed(0)
    model_path = Path("model.bin")
    if not model_path.exists():
        raise SystemExit("model.bin not found β€” run train.py first")

    params, chars, step, loss = load_model(model_path)
    print(f"Loaded: step={step}  loss={loss:.3f}")

    if len(sys.argv) > 1:
        print(generate(params, chars, sys.argv[1]))
    else:
        temperature, top_k = 0.8, 40
        print("Type a prompt.  :t 0.9 sets temperature, :k 20 sets top-k, empty line quits.")
        while True:
            try:
                line = input("> ")
            except (EOFError, KeyboardInterrupt):
                break
            if line.strip() == "" or line.strip() in ("exit", "quit"):
                break
            if line.startswith(":t "):
                temperature = float(line[3:]); continue
            if line.startswith(":k "):
                top_k = int(line[3:]); continue
            print(generate(params, chars, line, temperature=temperature, top_k=top_k))
            print()

build_corpus.py

build_corpus.py
#!/usr/bin/env python3
"""build_corpus.py β€” concatenate .md/.txt files into corpus.txt."""
import argparse, re
from pathlib import Path


def build_corpus(src, out):
    files = sorted(src.rglob("*")) if src.is_dir() else [src]
    parts = []
    for f in files:
        if not f.is_file() or f.suffix.lower() not in {".md", ".txt"}:
            continue
        text = f.read_text(encoding="utf-8")
        text = re.sub(r"[ \t]+", " ", text)
        text = re.sub(r"\n{3,}", "\n\n", text).strip()
        if text:
            parts.append(text)
    if not parts:
        raise SystemExit(f"No .md/.txt files found under {src}")
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text("\n\n".join(parts), encoding="utf-8")
    total = sum(len(p) for p in parts)
    print(f"Corpus: {out}  ({len(parts)} files, {total:,} chars)")


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("src", type=Path)
    ap.add_argument("out", type=Path)
    args = ap.parse_args()
    build_corpus(args.src, args.out)
Troubleshooting

Want to smoke-test first β†’ set STEPS = 50 (about 3–4 minutes in my container).

Loss stays near 4.65 (ln of the vocab size) β†’ the model isn't learning; check that LR and the corpus are what you think they are.

Output is repetitive β†’ raise temperature in the REPL: :t 0.9 or :t 1.0.

Output looks like word-shaped noise β†’ expected at this size and loss (~2.7). More steps, a bigger model, or a longer context all help, and all cost time.

Out of memory β†’ lower CTX_LEN or BATCH_SIZE.

That's the whole thing

Remember these two facts

1. Backpropagation is just the chain rule applied in reverse topological order. The autograd engine in Step 1 is the whole algorithm.

2. Transformer language models β€” from this tiny one to today's largest β€” are built from the same few primitives: matmul, add, softmax, a normalization layer and an activation. Newer models swap in variants (RMSNorm, SwiGLU, RoPE), but the structure is the same. Stack them, train them, sample from them.

You just built a language model with no libraries, no GPU, and no framework. Every operation is visible. That's the point: once you've seen the mechanism, the frameworks stop being magic.