Lesson 6 ยท Hands-on

Build a Real GPT in PyTorch

Lessons 3โ€“5 built, trained and sampled a 170K-parameter subword model in pure Python โ€” complete, but a toy. This lesson takes the same Transformer block and re-hosts it in PyTorch: 13.8M parameters, mixed-precision training, AdamW with cosine decay, checkpointing, and a real dataset (TinyStories). The concepts are the same. The engineering is what makes it work.

What stays

The Transformer block

Same pre-LN residual blocks, same causal self-attention, same GELU MLP. The math is identical to lessons 1โ€“3 โ€” only the tensor library changes.

What changes

Everything around it

micrograd โ†’ PyTorch autograd. Python loops โ†’ GPU tensors with FP16 mixed precision. ~1M tokens โ†’ ~7M tokens of TinyStories (BPE-8000). Lesson 4's validation split, warmup + cosine schedule and checkpoints return, re-implemented in PyTorch.

What it costs

PyTorch + a GPU

pip install torch. A full run wants a CUDA GPU (~15 minutes on a recent consumer card). CPU is fine for a smoke test, but the full 8K steps takes many hours.

micrograd โ†’ torch.nn ยท Python loops โ†’ GPU ยท 170K โ†’ 13.8M params same block, same loss, same objective โ€” now with fused kernels, AMP, and a real corpus
Begin โ†“

Why move to PyTorch โ€” what micrograd can't do

What you'll have by the end

Five Python files you can run, a trained 13.8M-parameter GPT checkpoint that writes short stories, and the ability to explain every part: the attention block, the data pipeline, the training loop (AdamW, warmup + cosine LR, mixed precision, gradient clipping, checkpoints) and sampling.

Before you start: lessons 1โ€“5 (the Transformer block from 1โ€“3; training from 4; sampling from 5), Python 3.9+, PyTorch 2.3+. Lessons 4 and 5 already explained validation, warmup + cosine, checkpoints, temperature and top-k, so here those get short recaps with pointers back. With a CUDA GPU the full run takes ~15 minutes; without one, follow along and run only the smoke test. Prefer to run first? Jump to Full source, run it, then read the steps to see why each piece exists.

The model from lessons 1โ€“5 works, but it's built on a 90-line autograd engine that does every operation in Python loops. That's fine for teaching โ€” it's not fine for a 10-million-parameter model. Three limits force the move:

  • Speed. micrograd's dot is a Python sum() over scalars. PyTorch's nn.Linear is a single BLAS call. On the same hardware, that's a 100โ€“1000ร— speedup before you even touch the GPU.
  • Memory. micrograd stores a Python object per scalar. Every parameter would be a Python Value object (13.8M of them), and every add or multiply creates more โ€” roughly a trillion scalar operations per training step at this size. Python cannot do that in any useful time.
  • Tooling. No fused attention, no mixed precision, no gradient checkpointing, no distributed training, no torch.compile. Every optimization modern training relies on is unavailable.

The move to PyTorch is not a change of model. It's a change of implementation. The TinyGPT class in architecture.py is the same block structure you already understand, written with nn.Module instead of hand-rolled Value arithmetic.

PropertyLessons 3โ€“5 (micrograd)Lesson 6 (PyTorch)
Parameters~170K~13.8M
Layers / heads / dim2 / 2 / 326 / 6 / 384
Context32 subwords256 subwords
Vocab2,256 (BPE)8,000 (BPE)
Data~95K tokens (L3) โ†’ ~1M (L4)~30 MB TinyStories + seed Q&A
Optimizermanual Adam + warmup/cosine (L4)AdamW + cosine + warmup
PrecisionFP64 Python floatsFP16 AMP
Training time~2โ€“3 h (CPU, L4 run)~15 min (GPU) / many hours (CPU)
Outputword fragmentsshort stories & Q&A answers
The scaling intuition, in one number

Loss scales roughly as a power law in parameters, data, and compute. Going from 170K โ†’ 13.8M parameters (81ร—) with ~7ร— more training tokens than lesson 4 doesn't make the model 81ร— better โ€” it moves it from "local word fragments" to "coherent short paragraphs." That's the qualitative gap this lesson crosses. Everything above this is the same curve, continued.

The architecture โ€” a pre-LN GPT, by the book

architecture.py is a standard decoder-only Transformer. It follows the GPT-2 layout: pre-LayerNorm residuals, GELU (tanh approximation), learned position embeddings, and weight tying between embedding and output head. One deliberate difference: the linear layers have no bias terms (GPT-2 had them; LLaMA-style models drop them, and little is lost). If you read the micrograd version, every line has a counterpart.

The building blocks, top to bottom

architecture.py โ€” attention
class CausalSelfAttention(nn.Module):
    def __init__(self, n_embd, n_head, dropout=0.0):
        super().__init__()
        assert n_embd % n_head == 0
        self.n_head = n_head
        self.head_dim = n_embd // n_head
        self.c_attn = nn.Linear(n_embd, 3 * n_embd, bias=False)
        self.c_proj = nn.Linear(n_embd, n_embd, bias=False)
        self.attn_drop = nn.Dropout(dropout)
        self.resid_drop = nn.Dropout(dropout)

    def forward(self, x):
        B, T, C = x.shape
        q, k, v = self.c_attn(x).split(C, dim=2)
        q = q.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        k = k.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        v = v.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        y = F.scaled_dot_product_attention(
            q, k, v,
            dropout_p=self.attn_drop.p if self.training else 0.0,
            is_causal=True,
        )
        y = y.transpose(1, 2).contiguous().view(B, T, C)
        return self.resid_drop(self.c_proj(y))

c_attn is one linear layer that produces Q, K, and V at once โ€” a 3 ร— n_embd output that gets split into three. Fusing them into one matrix multiply is faster than three separate nn.Linear calls and is what GPT-2 does.

The reshape dance view(B, T, n_head, head_dim).transpose(1, 2) turns (B, T, C) into (B, n_head, T, head_dim) so each head attends independently. This is the "multi-head" part โ€” the same Q/K/V math, run in parallel across n_head subspaces.

F.scaled_dot_product_attention is PyTorch 2.x's fused attention. On supported GPUs it dispatches to FlashAttention or a memory-efficient kernel; elsewhere it falls back to a standard implementation. Same math as the earlier lessons' manual softmax(q@kแต€/โˆšd)@v โ€” one call instead of ~20 lines, and faster because the full Tร—T score matrix need not be stored. is_causal=True applies the triangular mask inside the kernel, so no mask tensor needs to be materialized.

c_proj projects the concatenated heads back to n_embd. Both linear layers use bias=False: fewer parameters and a small speedup, with no noticeable quality loss at this scale.

architecture.py โ€” MLP and Block
class MLP(nn.Module):
    def __init__(self, n_embd, dropout=0.0):
        super().__init__()
        self.fc = nn.Linear(n_embd, 4 * n_embd, bias=False)
        self.proj = nn.Linear(4 * n_embd, n_embd, bias=False)
        self.drop = nn.Dropout(dropout)

    def forward(self, x):
        return self.drop(self.proj(F.gelu(self.fc(x), approximate="tanh")))


class Block(nn.Module):
    def __init__(self, n_embd, n_head, dropout=0.0):
        super().__init__()
        self.ln1 = nn.LayerNorm(n_embd)
        self.attn = CausalSelfAttention(n_embd, n_head, dropout)
        self.ln2 = nn.LayerNorm(n_embd)
        self.mlp = MLP(n_embd, dropout)

    def forward(self, x):
        x = x + self.attn(self.ln1(x))   # pre-LN residual
        x = x + self.mlp(self.ln2(x))    # pre-LN residual
        return x

The MLP expands n_embd โ†’ 4ยทn_embd, applies GELU, projects back. It's ~2ร— the parameter count of attention per block, and it's where most of the model's "knowledge" lives. The approximate="tanh" form of GELU is the variant GPT-2 used โ€” a tanh-based formula that closely tracks the exact GELU.

Pre-LN residuals: x = x + attn(ln(x)), not x = ln(x + attn(x)). The normalization happens before each sublayer, on the residual branch. This is what GPT-2 onward uses โ€” it keeps a clean gradient path through the residual stream, which makes deep stacks far easier to train than the original post-LN design (you still want LR warmup โ€” see step 4).

Why two LayerNorms per block? One before attention, one before the MLP. Each sublayer sees a normalized input, adds its output back to the unnormalized residual stream. The stream itself is never normalized until the final ln_f.

architecture.py โ€” TinyGPT
class TinyGPT(nn.Module):
    def __init__(self, vocab_size, n_layer=6, n_head=6, n_embd=384,
                 block_size=256, dropout=0.1):
        super().__init__()
        self.block_size = block_size
        self.tok_emb = nn.Embedding(vocab_size, n_embd)
        self.pos_emb = nn.Embedding(block_size, n_embd)
        self.drop = nn.Dropout(dropout)
        self.blocks = nn.ModuleList(
            [Block(n_embd, n_head, dropout) for _ in range(n_layer)]
        )
        self.ln_f = nn.LayerNorm(n_embd)
        self.head = nn.Linear(n_embd, vocab_size, bias=False)
        self.head.weight = self.tok_emb.weight  # weight tying

        self.apply(self._init_weights)
        for name, p in self.named_parameters():
            if name.endswith("proj.weight"):   # matches attn c_proj and mlp proj
                nn.init.normal_(p, mean=0.0, std=0.02 / math.sqrt(2 * n_layer))

Two embeddings, added. tok_emb maps token ids to vectors; pos_emb maps positions 0โ€ฆ255 to vectors. They're summed, then dropped out. Learned positional embeddings are simpler than RoPE (which modern models use) and work fine at this scale.

Weight tying: head.weight = tok_emb.weight. The input embedding and output projection share one vocab ร— n_embd matrix. This saves 3M parameters here, and empirically it improves small models โ€” the same vector that means "cat" on the way in also scores "cat" on the way out.

Scaled init for residual projections. The c_proj and MLP proj layers are initialized with std 0.02/โˆš(2ยทn_layer) instead of 0.02. This is GPT-2's trick: as depth grows, the residual stream accumulates variance from every block, so the layers that write into the stream are initialized smaller to compensate. It's why deep pre-LN Transformers train stably without architectural changes.

Parameter budget โ€” where the 13.8M goes

Token embedding (shared with the output head): 8000 ร— 384 = 3.07M. Position embedding: 256 ร— 384 = 0.10M.
Per block: attention 4 ร— 384ยฒ = 0.59M, MLP 8 ร— 384ยฒ = 1.18M, LayerNorms โ‰ˆ 0 โ†’ 1.77M. Six blocks: 10.63M.
Total = 3.07 + 0.10 + 10.63 โ‰ˆ 13.8M (exactly 13,797,120). Shares: MLPs 51%, attention 26%, embeddings 22%. Without weight tying it would be 16.9M.
Check it: your training log's [train] params: line should read 13.80M.

The forward pass and the loss

TinyGPT.forward(idx, targets) takes a (B, T) batch of token ids, runs them through embeddings โ†’ blocks โ†’ final LayerNorm โ†’ output head, and returns logits of shape (B, T, vocab_size). If targets is given, it also computes cross-entropy over the flattened (BยทT, vocab_size) logits against (BยทT,) targets โ€” exactly the next-token prediction objective from the earlier lessons, now in one F.cross_entropy call.

The data pipeline โ€” corpus, tokenizer, binary shards

Three scripts, four stages (below). build_corpus.py downloads 30 MB of TinyStories, adds a small seed Q&A set (repeated 50ร—), trains an 8K BPE tokenizer, encodes everything to a single uint16 array, and writes train.bin / val.bin with a 98/2 split. train.py memory-maps those shards. infer.py loads the tokenizer and the best checkpoint.

stage 1 text files โ†’ BPE tokenizer (vocab 8000) โ†’ tokenizer.json
stage 2 text files โ†’ encode with tokenizer โ†’ uint16 array
stage 3 uint16 array โ†’ 98/2 split โ†’ bin/train.bin, bin/val.bin
stage 4 train.py โ†’ np.memmap + fancy index โ†’ (B, T) batches

Why memory-map instead of loading?

A 30 MB corpus is ~7M tokens โ‰ˆ 14 MB as uint16 โ€” small enough to load. But the same pattern scales to a 30 GB corpus without changing a line: np.memmap pages in only the slices get_batch touches, and the OS page cache handles repetition. This is the exact pattern nanoGPT uses, and it's why the training loop never mentions the dataset size.

train.py โ€” MemmapDataset
class MemmapDataset:
    def __init__(self, path, block_size):
        self.data = np.memmap(path, dtype=np.uint16, mode="r")
        self.block_size = block_size

    def get_batch(self, batch_size):
        max_start = len(self.data) - self.block_size - 1
        ix = np.random.randint(0, max_start, batch_size)
        offsets = np.arange(self.block_size)
        x = self.data[ix[:, None] + offsets]
        y = self.data[ix[:, None] + offsets + 1]
        x = torch.from_numpy(x.astype(np.int64))
        y = torch.from_numpy(y.astype(np.int64))
        if DEVICE == "cuda":
            x = x.pin_memory().to(DEVICE, non_blocking=True)
            y = y.pin_memory().to(DEVICE, non_blocking=True)
        return x, y

np.memmap maps the binary file into virtual memory. Reading a slice data[a:b] triggers a page fault that loads only that page from disk โ€” the rest stays on disk. For training, the OS page cache keeps recently-touched pages resident, so repeated epochs over a small corpus are effectively RAM-speed.

The vectorized gather self.data[ix[:, None] + offsets] builds the whole (B, T) batch with one fancy-index instead of a Python loop over B. ix is (B,) random start positions; ix[:, None] + offsets broadcasts to (B, T) index matrix; indexing the memmap with it materializes (B, T) of uint16. y is the same slice shifted by one โ€” next-token prediction.

Why uint16? Vocab is 8,000, so token ids fit in 16 bits. Two bytes per token instead of eight is a 4ร— reduction in disk and memory bandwidth โ€” meaningful when the corpus is 30 GB.

Pinned memory + non_blocking=True: .pin_memory() puts the batch in page-locked RAM so the CPUโ†’GPU copy can run asynchronously. (Without pinning, non_blocking=True silently does nothing.) A small win here, bigger when data loading is the bottleneck.

The Q&A seed โ€” why it's there

TinyStories is all narrative. The seed block is 20 Q&A pairs in the format <|bos|>Q: โ€ฆ\nA: โ€ฆ<|eos|> โ€” each pair wrapped in its own start/end tokens. build_corpus.py repeats the seed 50ร— (about 1,000 examples, well under 1% of the data) so the model sees the format often enough to learn it. It won't become a Q&A system โ€” it effectively memorizes 20 answers โ€” but it learns to stay in format and to stop with <|eos|>. Story boundaries get the same treatment: TinyStories separates stories with the literal text <|endoftext|>, which the script rewrites to <|eos|><|bos|>, so stories and answers share one convention.

The 98/2 split, and why validation matters

The last 2% of the token stream is held out. Training loss tells you the model is fitting; validation loss tells you whether it's learning patterns that generalize. This run makes ~13 passes over its ~7M training tokens (see step 4), so some train/val gap is normal โ€” but the moment val rises while train keeps falling, you're overfitting. The training loop saves the best-val checkpoint separately for exactly this reason.

The training loop โ€” AdamW, cosine, AMP, checkpoints

Lesson 4 explained why you need validation, warmup + cosine, and resumable checkpoints; here they appear in PyTorch, plus two new ingredients: mixed precision (AMP) and AdamW. The loop is about 40 lines and contains the core techniques of modern LLM training. Read it as five layers: schedule, forward/backward, optimizer, evaluation, checkpoint.

train.py โ€” the loop, annotated
for it in range(start_iter, cfg.max_iters):
    # 1. LR schedule: linear warmup then cosine decay to min_lr
    lr = get_lr(it, cfg)
    for g in optimizer.param_groups:
        g["lr"] = lr

    # 2. forward under FP16 autocast (CUDA only)
    x, y = train_ds.get_batch(cfg.batch_size)
    with torch.amp.autocast(device_type="cuda", dtype=torch.float16,
                             enabled=(DEVICE == "cuda")):
        _, loss = model(x, y)

    # 3. scaled backward + grad clip + optimizer step
    scaler.scale(loss).backward()
    if cfg.grad_clip > 0:
        scaler.unscale_(optimizer)
        torch.nn.utils.clip_grad_norm_(model.parameters(), cfg.grad_clip)
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad(set_to_none=True)

    # 4. periodic eval on train + val
    if (it + 1) % cfg.eval_interval == 0 or it == cfg.max_iters - 1:
        metrics = estimate_loss(model, train_ds, val_ds, cfg)
        # 5. checkpoint resumable + best
        torch.save({"model": raw_model.state_dict(),
                    "optimizer": optimizer.state_dict(),
                    "scaler": scaler.state_dict(),
                    "iter": it, "best_val": min(best_val, metrics["val"]),
                    "config": asdict(cfg)}, ckpt_path)
        if metrics["val"] < best_val:
            best_val = metrics["val"]
            torch.save({"model": raw_model.state_dict(),
                        "config": asdict(cfg)}, "checkpoints/best.pt")

Warmup + cosine (get_lr): 300 steps of linear warmup from 0 to 3e-4, then cosine decay to 3e-5 over the remaining ~7,700 steps. Warmup prevents early divergence; cosine decay gives the model a long tail of small updates to settle into a good minimum.

AMP: autocast runs matmuls in FP16 and keeps losses/LayerNorms in FP32. GradScaler multiplies the loss by a dynamic factor before backward to prevent FP16 underflow, then unscales gradients before the optimizer step. On CPU both are no-ops (enabled=False) โ€” the code path is identical either way.

Grad clip: clip_grad_norm_(..., 1.0) rescales the gradient vector if its L2 norm exceeds 1.0. Cheap insurance against loss spikes; standard in every LLM training script.

Two checkpoints: ckpt.pt holds model + optimizer + scaler + iteration, so training resumes exactly where it stopped. best.pt holds only the weights that achieved the lowest val loss โ€” that's what infer.py loads.

The AdamW hyperparameters, and why each

SettingValueWhy
lr3e-4A safe, conservative choice for this size (small GPTs often go up to 1e-3). Too high diverges; too low underfits in 8K steps.
betas(0.9, 0.95)GPT-3's choice. ฮฒโ‚‚=0.95 adapts faster than the default 0.999 on short runs.
weight_decay0.1Decoupled (AdamW). Applied to weight matrices (2-D) only; LayerNorm gains are exempt.
grad_clip1.0Prevents occasional loss spikes from a bad batch.
warmup_iters300~4% of total. Enough to stabilize, short enough not to waste steps.
batch_size4848 ร— 256 tokens = 12,288 tokens/step. Fits comfortably in 4โ€“8 GB GPU memory under FP16.
max_iters80008000 ร— 12,288 โ‰ˆ 98M tokens seen โ€” about 13 passes over the ~7M-token corpus. Past this, extra passes mostly memorize rather than learn.

The LR schedule, drawn

warmup it โˆˆ [0, 300) lr = 3e-4 ยท (it+1)/300 โ† linear ramp
cosine it โˆˆ [300, 8000) lr = 3e-5 + ยฝ(1+cos(ฯ€ยทt))ยท(3e-4โˆ’3e-5) โ† t = progress 0โ†’1
floor it > 8000 lr = 3e-5 โ† never goes to 0

Why a floor instead of zero? Decaying to 10% of peak (min_lr = 3e-5) is the common recipe (GPT-3, Chinchilla and nanoGPT all use it), so the final steps still make small useful updates. The last line of the formula is only a safeguard: a normal run stops at step 8000 and never reaches it.

What the loss curve looks like

Random baseline: ln(8000) โ‰ˆ 8.99. Typical shape (your exact numbers will differ): the loss falls steeply over the first few hundred steps, then flattens slowly; expect roughly 2โ€“3 by step 8,000 (the reference value quoted in this lesson is ~2.9). Train and val should track each other early on. With ~13 passes over the data, a gap that opens later is the sign of overfitting โ€” if val rises while train keeps falling, stop (best.pt already holds your best weights) or add dropout.

Inference โ€” chat mode and top-k sampling

Lesson 5 covered temperature, top-k, top-p and repetition penalty in depth; this step only shows the minimal PyTorch version (temperature + top-k). infer.py loads best.pt and calls model.generate in one of two modes: story mode (--prompt "Once upon a time" โ€” your text is simply continued) or chat mode (run with no arguments โ€” each line you type is wrapped as Q: โ€ฆ\nA:). Both begin with <|bos|>, like the training data. The generation loop is the standard one: crop context to block_size, forward, temperature-scale logits, top-k filter, sample.

architecture.py โ€” generate()
@torch.no_grad()
def generate(self, idx, max_new_tokens, temperature=0.8, top_k=40, eos_id=None):
    for _ in range(max_new_tokens):
        idx_cond = idx[:, -self.block_size:]
        logits, _ = self(idx_cond)
        logits = logits[:, -1, :] / max(temperature, 1e-6)
        if top_k:
            v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
            logits[logits < v[:, [-1]]] = -float("inf")
        probs = F.softmax(logits, dim=-1)
        nxt = torch.multinomial(probs, num_samples=1)
        idx = torch.cat([idx, nxt], dim=1)
        if eos_id is not None and (nxt == eos_id).any():
            break
    return idx

@torch.no_grad disables autograd for the whole loop. Generation doesn't need gradients, and building a graph over hundreds of sequential steps would waste memory for nothing.

The context crop idx[:, -self.block_size:] is what lets generation run past 256 tokens at all: the position table has only 256 rows, so the model sees just the most recent 256 tokens and forgets everything earlier. (This simple loop re-runs the whole window for every new token; real systems cache keys/values to avoid that.)

Temperature divides logits before softmax. 0.5 sharpens the distribution (more deterministic); 1.2 flattens it (more random). 0.8 is the default sweet spot for a small model.

Top-k keeps only the k highest-probability tokens and sets the rest to -inf, so softmax gives them zero probability. This stops the model from occasionally sampling a very unlikely token and derailing the sentence โ€” a common failure for small models.

EOS early stop: (nxt == eos_id).any() stops generation as soon as a sequence emits <|eos|> (written for batch size 1, which is how infer.py uses it). In chat mode the model has learned to end its answer with EOS, so this actually fires.

Wrapping the prompt

The model was trained on text shaped <|bos|>Q: โ€ฆ\nA: โ€ฆ<|eos|>. In chat mode, infer.py builds <|bos|>Q: {prompt}\nA: and lets generate continue. Because every seed answer ended with <|eos|> in training, the model learns to emit it when the answer is done, and generation stops. Story mode skips the Q:/A: wrapper, so the model just continues your text as a story.

Example output (illustrative โ€” sampling is random, yours will differ)

terminal โ€” both modes
$ python infer.py --prompt "Once upon a time"
Once upon a time, there was a little girl named Lily. She loved to
play in the park with her friends. One day, she found a shiny red ball...

$ python infer.py
Q: What is Python?
A: Python is a high-level, interpreted programming language known for readability.

Q: What is the capital of Spain?
A: The capital of France is Paris.     <- format learned, facts not

It's not GPT-4. Story mode gives coherent multi-sentence English โ€” a qualitative jump from the word fragments of lessons 3โ€“5. Chat mode only "knows" the 20 seed answers; ask anything else and it falls back to a story or a wrong seed answer. The format is learned, the facts are not โ€” which is what 30 MB of children's stories plus a tiny format seed can do.

Things to try

Temperature sweep: run the same prompt at --temp 0.3, 0.8, 1.3. Watch coherence collapse into word salad at the top end.

Top-k ablation: set --top_k 0 (no filter) and compare. The long tail is exactly what top-k removes.

Resume test: checkpoints are written every 500 steps. Press Ctrl-C shortly after one is saved, then rerun python train.py. The log should say resumed: next step is โ€ฆ and the loss should continue from about where it was โ€” not restart near 9.

Overfit deliberately: set --max_iters 40000 and watch val loss rise while train loss falls. At that length the model makes ~65 passes over 30 MB and starts memorizing โ€” the failure the 8K default is chosen to limit.

What you actually get โ€” and what's still missing

This is the first model in the series that produces text a human would call "writing." Here's the honest scorecard.

13.8M
Parameters
~2.9
Val loss
256
Context tokens
~15m
GPU train time

Val loss and time are approximate reference values; they depend on your GPU, seed and corpus slice.

What works

  • Coherent paragraphs. Multi-sentence stories with consistent characters and plots that hold together for 3โ€“5 sentences.
  • Format adherence. Q&A prompts get Q&A-shaped answers; story prompts get stories. The seed data did its job.
  • No unknowns. Byte-level BPE from lesson 3 carries over โ€” rare words, names, typos all tokenize.
  • Resumable. Checkpoints survive interruption, and seeds are fixed, so runs are close to repeatable (GPU kernels aren't bit-exact, so don't expect identical digits).

What's missing โ€” the honest gaps

  • Factual knowledge. ~30 MB of children's stories contains almost no facts. It will happily answer "What is the capital of France?" with a story about a girl named Lily.
  • Long-range coherence. 256 tokens is only ~190 words. The model sees just its most recent 256 tokens, so in a longer story it forgets how it began.
  • Instruction following. It continues text in a style; it doesn't follow instructions. RLHF is what turns a text-continuer into an assistant.
  • Reasoning. No chain-of-thought, no tool use, no math beyond memorized seed examples. Those are all scale + post-training, not architecture.
ModelParamsDataContextWhat it can do
Lessons 3โ€“5 (micrograd)170K~1M tokens32word fragments
Lesson 6 (this)13.8M30 MB256short coherent stories
GPT-2 small124M40 GB1024coherent essays
GPT-3175B~570 GB2048few-shot reasoning
GPT-4undisclosedundisclosed8Kโ€“128K (by version)broad competence

The gap from here to GPT-3 is already ~12,000ร— in parameters and ~19,000ร— in data (GPT-4's numbers aren't public). The architecture barely changes. What changes is scale, data quality, and post-training (SFT + RLHF). Those are the next lessons โ€” and the reason this one matters is that everything above is the same block, trained longer.

The real lesson: engineering is the multiplier

The lesson 4โ€“5 model and this one share the same architecture, the same loss, the same objective. What separates them is engineering: PyTorch's fused kernels, mixed precision, AdamW with a real schedule, a real dataset, and resumable checkpoints. The math was done in lessons 1โ€“5. This lesson is about the 90% of ML work that isn't math โ€” and about how much a 81ร— parameter increase, done right, actually buys you.

Where this leads

You now have a real, working GPT. The path forward is three independent knobs, and none of them require a new concept:

KnobChangeEffectCost
Model sizen_layer 6โ†’12, n_embd 384โ†’768, n_head 6โ†’12~6.6ร— params, sharper local coherence~7ร— compute
DataTinyStories โ†’ OpenWebText / FineWebfactual knowledge, world models100ร— storage, tokenizer retrain
Post-trainingSFT on instructions โ†’ RLHFturns text-continuer into assistantnew pipeline, human labels

A good next experiment is data: swap TinyStories for a 1โ€“2 GB slice of a cleaned web corpus (for example FineWeb-Edu), retrain the tokenizer (keep vocab โ‰ค 65,535 so uint16 still works; 16Kโ€“32K suits web text), and train for ~30K steps. Expect more varied, fluent text โ€” but a model this small still stores few reliable facts. Facts need more parameters and more data together.

The full source

To run it

Requires: Python 3.9+, PyTorch 2.3+ (the code uses torch.amp.GradScaler), plus numpy and tokenizers. A CUDA GPU is strongly recommended: a full CPU run takes many hours (roughly 10 h or more), so on CPU do the smoke test only.

Layout: five files in one folder โ€” architecture.py, tokenizer.py, build_corpus.py, train.py, infer.py (full code below). Then run:

pip install torch numpy tokenizers
python build_corpus.py          # ~2 min: download + tokenize + shard
python train.py                 # ~15 min on a modern GPU
python infer.py --prompt "Once upon a time"   # story mode
python infer.py                 # chat mode (type a question)

Check your run: build_corpus.py prints vocab_size = 8000 and roughly 7M train tokens; train.py prints params: 13.80M; the first logged loss is near 9 (ln 8000 = 8.99) and falls fast; val loss ends roughly in the 2โ€“3 range (reference โ‰ˆ 2.9); peak VRAM is a few GB.

What about micrograd.py?

It's not here โ€” this lesson replaces it entirely. PyTorch's autograd engine does everything micrograd.py did, faster and on tensors. If you want the from-scratch version, it's in lessons 1โ€“5. The five files below are the complete project.

architecture.py โ€” complete

architecture.py
import math
import torch
import torch.nn as nn
import torch.nn.functional as F


class CausalSelfAttention(nn.Module):
    def __init__(self, n_embd, n_head, dropout=0.0):
        super().__init__()
        assert n_embd % n_head == 0
        self.n_head = n_head
        self.head_dim = n_embd // n_head
        self.c_attn = nn.Linear(n_embd, 3 * n_embd, bias=False)
        self.c_proj = nn.Linear(n_embd, n_embd, bias=False)
        self.attn_drop = nn.Dropout(dropout)
        self.resid_drop = nn.Dropout(dropout)

    def forward(self, x):
        B, T, C = x.shape
        q, k, v = self.c_attn(x).split(C, dim=2)
        q = q.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        k = k.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        v = v.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
        y = F.scaled_dot_product_attention(
            q, k, v,
            dropout_p=self.attn_drop.p if self.training else 0.0,
            is_causal=True,
        )
        y = y.transpose(1, 2).contiguous().view(B, T, C)
        return self.resid_drop(self.c_proj(y))


class MLP(nn.Module):
    def __init__(self, n_embd, dropout=0.0):
        super().__init__()
        self.fc = nn.Linear(n_embd, 4 * n_embd, bias=False)
        self.proj = nn.Linear(4 * n_embd, n_embd, bias=False)
        self.drop = nn.Dropout(dropout)

    def forward(self, x):
        return self.drop(self.proj(F.gelu(self.fc(x), approximate="tanh")))


class Block(nn.Module):
    def __init__(self, n_embd, n_head, dropout=0.0):
        super().__init__()
        self.ln1 = nn.LayerNorm(n_embd)
        self.attn = CausalSelfAttention(n_embd, n_head, dropout)
        self.ln2 = nn.LayerNorm(n_embd)
        self.mlp = MLP(n_embd, dropout)

    def forward(self, x):
        x = x + self.attn(self.ln1(x))
        x = x + self.mlp(self.ln2(x))
        return x


class TinyGPT(nn.Module):
    def __init__(self, vocab_size, n_layer=6, n_head=6, n_embd=384,
                 block_size=256, dropout=0.1):
        super().__init__()
        self.block_size = block_size
        self.tok_emb = nn.Embedding(vocab_size, n_embd)
        self.pos_emb = nn.Embedding(block_size, n_embd)
        self.drop = nn.Dropout(dropout)
        self.blocks = nn.ModuleList(
            [Block(n_embd, n_head, dropout) for _ in range(n_layer)]
        )
        self.ln_f = nn.LayerNorm(n_embd)
        self.head = nn.Linear(n_embd, vocab_size, bias=False)
        self.head.weight = self.tok_emb.weight  # weight tying

        self.apply(self._init_weights)
        # scaled init for residual projections
        for name, p in self.named_parameters():
            if name.endswith("proj.weight"):   # matches attn c_proj and mlp proj
                nn.init.normal_(p, mean=0.0, std=0.02 / math.sqrt(2 * n_layer))

    @staticmethod
    def _init_weights(m):
        if isinstance(m, nn.Linear):
            nn.init.normal_(m.weight, mean=0.0, std=0.02)
            if m.bias is not None:
                nn.init.zeros_(m.bias)
        elif isinstance(m, nn.Embedding):
            nn.init.normal_(m.weight, mean=0.0, std=0.02)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        assert T <= self.block_size
        pos = torch.arange(T, device=idx.device)
        x = self.drop(self.tok_emb(idx) + self.pos_emb(pos))
        for blk in self.blocks:
            x = blk(x)
        x = self.ln_f(x)
        logits = self.head(x)
        loss = None
        if targets is not None:
            loss = F.cross_entropy(
                logits.view(-1, logits.size(-1)), targets.view(-1), ignore_index=-1
            )
        return logits, loss

    @torch.no_grad()
    def generate(self, idx, max_new_tokens, temperature=0.8, top_k=40,
                 eos_id=None):
        for _ in range(max_new_tokens):
            idx_cond = idx[:, -self.block_size:]
            logits, _ = self(idx_cond)
            logits = logits[:, -1, :] / max(temperature, 1e-6)
            if top_k:
                v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
                logits[logits < v[:, [-1]]] = -float("inf")
            probs = F.softmax(logits, dim=-1)
            nxt = torch.multinomial(probs, num_samples=1)
            idx = torch.cat([idx, nxt], dim=1)
            if eos_id is not None and (nxt == eos_id).any():
                break
        return idx

tokenizer.py โ€” complete

tokenizer.py
"""
Byte-level BPE tokenizer.
"""
import os

try:
    from tokenizers import Tokenizer
    from tokenizers.models import BPE
    from tokenizers.trainers import BpeTrainer
    from tokenizers.pre_tokenizers import ByteLevel
    from tokenizers.decoders import ByteLevel as ByteLevelDecoder
except ImportError as e:
    raise SystemExit("Missing dependency - run: pip install tokenizers") from e


class BPETokenizer:
    def __init__(self, tok: "Tokenizer"):
        self.tok = tok
        self.pad_id = tok.token_to_id("<|pad|>")
        self.bos_id = tok.token_to_id("<|bos|>")
        self.eos_id = tok.token_to_id("<|eos|>")
        self.vocab_size = tok.get_vocab_size()

    def encode(self, text, add_special=True):
        ids = self.tok.encode(text).ids
        if add_special:
            ids = [self.bos_id] + ids + [self.eos_id]
        return ids

    def decode(self, ids):
        return self.tok.decode(ids, skip_special_tokens=True)

    def save(self, path):
        self.tok.save(path)

    @classmethod
    def load(cls, path):
        return cls(Tokenizer.from_file(path))


def train_tokenizer(text_files, vocab_size=8000, out_path="tokenizer.json"):
    tok = Tokenizer(BPE(unk_token=None))
    tok.pre_tokenizer = ByteLevel(add_prefix_space=False)
    tok.decoder = ByteLevelDecoder()
    trainer = BpeTrainer(
        vocab_size=vocab_size,
        special_tokens=["<|pad|>", "<|bos|>", "<|eos|>"],
        initial_alphabet=ByteLevel.alphabet(),
        show_progress=True,
    )
    tok.train(text_files, trainer)
    tok.save(out_path)
    return BPETokenizer(tok)


def load_tokenizer(path="tokenizer.json"):
    if not os.path.exists(path):
        raise FileNotFoundError(f"{path} not found โ€” run build_corpus.py first")
    return BPETokenizer.load(path)

build_corpus.py โ€” complete

build_corpus.py
"""
Build corpus -> train tokenizer -> write train.bin/val.bin
"""
import os
import sys
import glob
import re
import urllib.request

import numpy as np

from tokenizer import train_tokenizer, BPETokenizer

DATA_DIR = "data"
TOK_PATH = "tokenizer.json"
OUT_DIR = "bin"
SEED_REPEAT = 50   # the 20 seed pairs are tiny; repeat them so the format is learned

# -------- seed Q&A (format: <|bos|>Q: ...\nA: ...<|eos|>) --------
SEED_QA = """\
Q: What is the capital of France?
A: The capital of France is Paris.
Q: What is 2 + 2?
A: 2 + 2 equals 4.
Q: What color is the sky on a clear day?
A: On a clear day the sky is blue.
Q: Who wrote Romeo and Juliet?
A: William Shakespeare wrote Romeo and Juliet.
Q: What is the boiling point of water at sea level?
A: Water boils at 100 degrees Celsius (212 Fahrenheit) at sea level.
Q: What does CPU stand for?
A: CPU stands for Central Processing Unit.
Q: What does GPU stand for?
A: GPU stands for Graphics Processing Unit.
Q: What is Python?
A: Python is a high-level, interpreted programming language known for readability.
Q: What is machine learning?
A: Machine learning is a field of AI where models learn patterns from data.
Q: What is a neural network?
A: A neural network is a set of layers of weighted connections trained to map inputs to outputs.
Q: How many days are in a week?
A: There are seven days in a week.
Q: How many hours are in a day?
A: There are 24 hours in a day.
Q: What is the largest planet in our solar system?
A: Jupiter is the largest planet in our solar system.
Q: What is the smallest prime number?
A: The smallest prime number is 2.
Q: What is the square root of 16?
A: The square root of 16 is 4.
Q: What language is used to write this program?
A: This program is written in Python.
Q: What is 10 multiplied by 5?
A: 10 multiplied by 5 equals 50.
Q: Name a primary color.
A: Red, blue, and yellow are primary colors.
Q: What is the freezing point of water?
A: Water freezes at 0 degrees Celsius (32 Fahrenheit).
Q: What is the speed of light?
A: Light travels at about 299,792,458 meters per second in a vacuum.
"""


def download_tinystories(max_mb=30):
    """Grab a chunk of TinyStories (small, clean, fast to fit)."""
    dst = os.path.join(DATA_DIR, "tinystories.txt")
    if os.path.exists(dst) and os.path.getsize(dst) > 1_000_000:
        return dst
    url = "https://huggingface.co/datasets/roneneldan/TinyStories/resolve/main/TinyStoriesV2-GPT4-train.txt"
    print(f"[corpus] downloading TinyStories (~{max_mb} MB)...")
    try:
        req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
        with urllib.request.urlopen(req, timeout=120) as r, open(dst, "wb") as f:
            got = 0
            limit = max_mb * 1024 * 1024
            while got < limit:
                chunk = r.read(1024 * 256)
                if not chunk:
                    break
                f.write(chunk)
                got += len(chunk)
        print(f"[corpus] saved {got/1e6:.1f} MB -> {dst}")
        return dst
    except Exception as e:
        print(f"[corpus] download failed ({e}); using seed only.")
        return None


def format_qa(block):
    """Wrap Q&A into training format."""
    pairs = re.split(r"\n(?=Q: )", block.strip())   # one chunk per Q/A pair
    return "\n".join("<|bos|>" + p.strip() + "<|eos|>" for p in pairs)


def main():
    os.makedirs(DATA_DIR, exist_ok=True)
    os.makedirs(OUT_DIR, exist_ok=True)

    qa_path = os.path.join(DATA_DIR, "qa.txt")
    with open(qa_path, "w", encoding="utf-8") as f:
        f.write("\n".join([format_qa(SEED_QA)] * SEED_REPEAT))
    print(f"[corpus] wrote seed Q&A -> {qa_path}")

    story_path = download_tinystories(max_mb=30)
    sources = [qa_path] + ([story_path] if story_path else [])

    # train tokenizer
    if not os.path.exists(TOK_PATH):
        print(f"[corpus] training tokenizer (vocab 8000)...")
        train_tokenizer(sources, vocab_size=8000, out_path=TOK_PATH)
        print(f"[corpus] tokenizer saved -> {TOK_PATH}")

    # load tokenizer
    tok = BPETokenizer.load(TOK_PATH)
    print(f"[corpus] tokenizer vocab_size = {tok.vocab_size}")

    # encode all text into one big array
    all_ids = []
    for p in sources:
        print(f"[corpus] encoding {p} ...")
        with open(p, "r", encoding="utf-8", errors="ignore") as f:
            text = f.read()
        text = text.replace("<|endoftext|>", "<|eos|><|bos|>")   # TinyStories story separator
        all_ids.extend(tok.encode(text, add_special=False))
    assert max(all_ids) < 65536, "vocab too large for uint16 โ€” use uint32"
    arr = np.array(all_ids, dtype=np.uint16)
    print(f"[corpus] total tokens: {len(arr):,}")

    # split 98 / 2
    n = len(arr)
    split = int(n * 0.98)
    train, val = arr[:split], arr[split:]
    train.tofile(os.path.join(OUT_DIR, "train.bin"))
    val.tofile(os.path.join(OUT_DIR, "val.bin"))
    print(f"[corpus] train tokens: {len(train):,}  val tokens: {len(val):,}")
    print(f"[corpus] done.")


if __name__ == "__main__":
    main()

train.py โ€” complete

train.py
import os
import math
import time
import argparse
from dataclasses import dataclass, asdict

import numpy as np
import torch

from architecture import TinyGPT
from tokenizer import BPETokenizer

# ---------------- SETUP ----------------
torch.manual_seed(1337)   # fixes init + dropout; batches use np.random below
np.random.seed(1337)

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"


# ---------------- CONFIG ----------------
@dataclass
class Config:
    data_dir: str = "bin"
    ckpt_dir: str = "checkpoints"
    tok_path: str = "tokenizer.json"
    block_size: int = 256
    batch_size: int = 48
    n_layer: int = 6
    n_head: int = 6
    n_embd: int = 384
    dropout: float = 0.1
    lr: float = 3e-4
    min_lr: float = 3e-5
    weight_decay: float = 0.1
    beta1: float = 0.9
    beta2: float = 0.95
    warmup_iters: int = 300
    max_iters: int = 8000
    eval_interval: int = 500
    eval_iters: int = 100
    log_interval: int = 20
    grad_clip: float = 1.0
    use_compile: bool = False
    vocab_size: int = 0        # filled in from the tokenizer at startup


# ---------------- DATA ----------------
class MemmapDataset:
    def __init__(self, path, block_size):
        if not os.path.exists(path):
            raise SystemExit(
                f"[train] {path} not found.\n"
                f"        Run `python build_corpus.py` from "
                f"{os.path.dirname(os.path.abspath(__file__))} first."
            )
        self.data = np.memmap(path, dtype=np.uint16, mode="r")
        self.block_size = block_size

    def get_batch(self, batch_size):
        max_start = len(self.data) - self.block_size - 1
        assert max_start >= 1, f"corpus too small ({len(self.data)} tokens)"
        ix = np.random.randint(0, max_start, batch_size)
        offsets = np.arange(self.block_size)
        x = self.data[ix[:, None] + offsets]
        y = self.data[ix[:, None] + offsets + 1]
        x = torch.from_numpy(x.astype(np.int64))
        y = torch.from_numpy(y.astype(np.int64))
        if DEVICE == "cuda":
            x = x.pin_memory().to(DEVICE, non_blocking=True)
            y = y.pin_memory().to(DEVICE, non_blocking=True)
        return x, y


# ---------------- LR SCHEDULE ----------------
def get_lr(it, cfg):
    if it < cfg.warmup_iters:
        return cfg.lr * (it + 1) / cfg.warmup_iters
    if it > cfg.max_iters:
        return cfg.min_lr
    ratio = (it - cfg.warmup_iters) / (cfg.max_iters - cfg.warmup_iters)
    coeff = 0.5 * (1.0 + math.cos(math.pi * ratio))
    return cfg.min_lr + coeff * (cfg.lr - cfg.min_lr)


# ---------------- EVAL ----------------
@torch.no_grad()
def estimate_loss(model, train_ds, val_ds, cfg):
    model.eval()
    out = {}
    for name, ds in [("train", train_ds), ("val", val_ds)]:
        losses = torch.zeros(cfg.eval_iters)
        for k in range(cfg.eval_iters):
            x, y = ds.get_batch(cfg.batch_size)
            with torch.amp.autocast(device_type="cuda", dtype=torch.float16,
                                     enabled=(DEVICE == "cuda")):
                _, loss = model(x, y)
            losses[k] = loss.item()
        out[name] = losses.mean().item()
    model.train()
    return out


# ---------------- MAIN ----------------
def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--max_iters", type=int, default=None)
    ap.add_argument("--batch_size", type=int, default=None)
    ap.add_argument("--block_size", type=int, default=None)
    ap.add_argument("--n_layer", type=int, default=None)
    ap.add_argument("--n_embd", type=int, default=None)
    ap.add_argument("--n_head", type=int, default=None)
    ap.add_argument("--compile", action="store_true",
                    help="enable torch.compile (requires triton on Windows+CUDA)")
    args = ap.parse_args()

    cfg = Config()
    if args.max_iters is not None:  cfg.max_iters = args.max_iters
    if args.batch_size is not None: cfg.batch_size = args.batch_size
    if args.block_size is not None: cfg.block_size = args.block_size
    if args.n_layer is not None:    cfg.n_layer = args.n_layer
    if args.n_embd is not None:     cfg.n_embd = args.n_embd
    if args.n_head is not None:     cfg.n_head = args.n_head
    if args.compile:                cfg.use_compile = True

    os.makedirs(cfg.ckpt_dir, exist_ok=True)

    # Tokenizer vocab
    tok = BPETokenizer.load(cfg.tok_path)
    cfg.vocab_size = tok.vocab_size
    print(f"[train] device={DEVICE} vocab={cfg.vocab_size}")

    train_ds = MemmapDataset(os.path.join(cfg.data_dir, "train.bin"), cfg.block_size)
    val_ds = MemmapDataset(os.path.join(cfg.data_dir, "val.bin"), cfg.block_size)

    model = TinyGPT(
        vocab_size=cfg.vocab_size,
        n_layer=cfg.n_layer,
        n_head=cfg.n_head,
        n_embd=cfg.n_embd,
        block_size=cfg.block_size,
        dropout=cfg.dropout,
    ).to(DEVICE)
    n_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
    print(f"[train] params: {n_params/1e6:.2f}M")

    raw_model = model                       # un-compiled handle, used for save/load
    if cfg.use_compile and hasattr(torch, "compile"):
        model = torch.compile(model)
        print("[train] torch.compile enabled")

    # weight decay on matrices (2-D) only, not on LayerNorm gains
    decay = [p for p in raw_model.parameters() if p.dim() >= 2]
    no_decay = [p for p in raw_model.parameters() if p.dim() < 2]
    optimizer = torch.optim.AdamW(
        [{"params": decay, "weight_decay": cfg.weight_decay},
         {"params": no_decay, "weight_decay": 0.0}],
        lr=cfg.lr, betas=(cfg.beta1, cfg.beta2),
    )
    scaler = torch.amp.GradScaler(device="cuda", enabled=(DEVICE == "cuda"))

    # ---- resume if checkpoint exists ----
    start_iter = 0
    best_val = float("inf")
    ckpt_path = os.path.join(cfg.ckpt_dir, "ckpt.pt")
    if os.path.exists(ckpt_path):
        ck = torch.load(ckpt_path, map_location=DEVICE, weights_only=True)
        try:
            raw_model.load_state_dict(ck["model"])
        except RuntimeError as e:
            raise SystemExit(f"[train] {ckpt_path} doesn't match this config ({e}).\n"
                             f"        Delete the checkpoints/ folder to start fresh.")
        optimizer.load_state_dict(ck["optimizer"])
        scaler.load_state_dict(ck["scaler"])
        start_iter = ck["iter"] + 1
        best_val = ck.get("best_val", best_val)
        if start_iter >= cfg.max_iters:
            raise SystemExit("[train] this run already finished. Delete checkpoints/ to retrain.")
        print(f"[train] resumed: next step is {start_iter + 1} of {cfg.max_iters}")

    model.train()
    t0 = time.time()
    running = None
    for it in range(start_iter, cfg.max_iters):
        lr = get_lr(it, cfg)
        for g in optimizer.param_groups:
            g["lr"] = lr

        x, y = train_ds.get_batch(cfg.batch_size)
        with torch.amp.autocast(device_type="cuda", dtype=torch.float16,
                                 enabled=(DEVICE == "cuda")):
            _, loss = model(x, y)
        scaler.scale(loss).backward()
        if cfg.grad_clip > 0:
            scaler.unscale_(optimizer)
            torch.nn.utils.clip_grad_norm_(model.parameters(), cfg.grad_clip)
        scaler.step(optimizer)
        scaler.update()
        optimizer.zero_grad(set_to_none=True)

        running = loss.item() if running is None else 0.95 * running + 0.05 * loss.item()

        if (it + 1) % cfg.log_interval == 0:
            dt = max(time.time() - t0, 1e-9)
            tok_s = (it + 1 - start_iter) * cfg.batch_size * cfg.block_size / dt
            print(f"iter {it+1:5d} | loss {running:.4f} | lr {lr:.2e} | {tok_s/1e3:.1f}k tok/s")

        if (it + 1) % cfg.eval_interval == 0 or it == cfg.max_iters - 1:
            metrics = estimate_loss(model, train_ds, val_ds, cfg)
            print(f"[eval] iter {it+1}: train {metrics['train']:.4f} | val {metrics['val']:.4f}")
            ck = {
                "model": raw_model.state_dict(),
                "optimizer": optimizer.state_dict(),
                "scaler": scaler.state_dict(),
                "iter": it,
                "best_val": min(best_val, metrics["val"]),
                "config": asdict(cfg),
            }
            torch.save(ck, ckpt_path)
            if metrics["val"] < best_val:
                best_val = metrics["val"]
                torch.save({"model": raw_model.state_dict(), "config": asdict(cfg)},
                           os.path.join(cfg.ckpt_dir, "best.pt"))
            print(f"[train] checkpoint saved -> {ckpt_path}")

    print("[train] done.")


if __name__ == "__main__":
    main()

infer.py โ€” complete

infer.py
import argparse
import os

import torch

from architecture import TinyGPT
from tokenizer import BPETokenizer

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
CKPT = "checkpoints/best.pt"
TOK = "tokenizer.json"


def load_model():
    tok = BPETokenizer.load(TOK)

    ck = torch.load(CKPT, map_location=DEVICE, weights_only=True)
    cfg = ck["config"]
    model = TinyGPT(
        vocab_size=tok.vocab_size,
        n_layer=cfg["n_layer"],
        n_head=cfg["n_head"],
        n_embd=cfg["n_embd"],
        block_size=cfg["block_size"],
        dropout=0.0,
    ).to(DEVICE)
    model.load_state_dict(ck["model"])
    model.eval()
    return model, tok


def generate(model, tok, prompt, max_new=200, temperature=0.8, top_k=40):
    full = f"<|bos|>{prompt.strip()}"
    ids = tok.encode(full, add_special=False)
    if not ids or ids[0] != tok.bos_id:
        ids = [tok.bos_id] + ids
    x = torch.tensor(ids, dtype=torch.long, device=DEVICE).unsqueeze(0)
    with torch.amp.autocast(device_type="cuda", dtype=torch.float16, enabled=(DEVICE == "cuda")):
        out = model.generate(
            x, max_new_tokens=max_new,
            temperature=temperature, top_k=top_k, eos_id=tok.eos_id,
        )
    text = tok.decode(out[0].tolist())
    return text


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--prompt", type=str, default=None,
                    help="story mode: continue this text. Omit for chat mode.")
    ap.add_argument("--temp", type=float, default=0.8)
    ap.add_argument("--top_k", type=int, default=40)
    ap.add_argument("--max_new", type=int, default=200)
    args = ap.parse_args()

    model, tok = load_model()
    print(f"[infer] loaded ({DEVICE})")

    if args.prompt:
        print(generate(model, tok, args.prompt, args.max_new, args.temp, args.top_k))
        return

    print("Chat mode. Type 'quit' to exit.\n")
    while True:
        try:
            prompt = input("Q: ").strip()
        except (EOFError, KeyboardInterrupt):
            print()
            break
        if prompt.lower() in {"quit", "exit", "q"}:
            break
        if not prompt:
            continue
        wrapped = f"Q: {prompt}\nA:"
        out = generate(model, tok, wrapped, args.max_new, args.temp, args.top_k)
        out = out.replace("<|bos|>", "").replace("<|eos|>", "")
        print(f"A:{out.split('A:', 1)[-1]}\n")


if __name__ == "__main__":
    main()
Troubleshooting

Smoke test first โ†’ --max_iters 20 --batch_size 4. Everything should finish in under a minute. Then delete the checkpoints/ folder โ€” otherwise the real run resumes from the smoke test's weights.

CUDA out of memory โ†’ lower --batch_size to 24 or 12. The model is small; batch is the knob.

Loss stuck at 8.99 โ†’ LR too high or warmup too short. Check get_lr(0) returns a small positive number.

Tokenizer mismatch on resume โ†’ tokenizer.json was regenerated between runs. Don't delete it while training.

torch.compile fails on Windows โ†’ expected; it needs triton. Leave --compile off.

AttributeError: torch.amp has no GradScaler โ†’ PyTorch is older than 2.3. Upgrade it.

"this run already finished" โ†’ a completed ckpt.pt exists. Delete checkpoints/ to retrain from scratch.

What you just built

Three takeaways

1. The architecture didn't change. Same pre-LN Transformer block as lessons 1โ€“3 โ€” PyTorch just runs it on tensors instead of scalars. The move to a real framework is an engineering upgrade, not a modeling one.

2. The training loop is where the magic is. AdamW + cosine + warmup + AMP + grad clip + checkpointing โ€” that's ~40 lines and it's the difference between "trains" and "trains well." Most production LLM training loops have this same shape.

3. Data and scale are the next frontier. 13.8M params on 30 MB gives coherent stories. 124M on 40 GB gives essays. The block is the same; the multiplier is scale.

You now have the whole path: character, word and subword models, a proper training loop, sampling strategies, and a real PyTorch GPT โ€” all built from the same block. Natural next steps are the three knobs above: scale the model, swap in broader data, and add post-training. Lesson 5's top-p and repetition penalty also drop straight into generate(), as a good exercise.