2
The architecture โ a pre-LN GPT, by the book
architecture.py is a standard decoder-only Transformer. It follows the GPT-2 layout: pre-LayerNorm residuals, GELU (tanh approximation), learned position embeddings, and weight tying between embedding and output head. One deliberate difference: the linear layers have no bias terms (GPT-2 had them; LLaMA-style models drop them, and little is lost). If you read the micrograd version, every line has a counterpart.
The building blocks, top to bottom
class CausalSelfAttention(nn.Module):
def __init__(self, n_embd, n_head, dropout=0.0):
super().__init__()
assert n_embd % n_head == 0
self.n_head = n_head
self.head_dim = n_embd // n_head
self.c_attn = nn.Linear(n_embd, 3 * n_embd, bias=False)
self.c_proj = nn.Linear(n_embd, n_embd, bias=False)
self.attn_drop = nn.Dropout(dropout)
self.resid_drop = nn.Dropout(dropout)
def forward(self, x):
B, T, C = x.shape
q, k, v = self.c_attn(x).split(C, dim=2)
q = q.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
k = k.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
v = v.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
y = F.scaled_dot_product_attention(
q, k, v,
dropout_p=self.attn_drop.p if self.training else 0.0,
is_causal=True,
)
y = y.transpose(1, 2).contiguous().view(B, T, C)
return self.resid_drop(self.c_proj(y))
c_attn is one linear layer that produces Q, K, and V at once โ a 3 ร n_embd output that gets split into three. Fusing them into one matrix multiply is faster than three separate nn.Linear calls and is what GPT-2 does.
The reshape dance view(B, T, n_head, head_dim).transpose(1, 2) turns (B, T, C) into (B, n_head, T, head_dim) so each head attends independently. This is the "multi-head" part โ the same Q/K/V math, run in parallel across n_head subspaces.
F.scaled_dot_product_attention is PyTorch 2.x's fused attention. On supported GPUs it dispatches to FlashAttention or a memory-efficient kernel; elsewhere it falls back to a standard implementation. Same math as the earlier lessons' manual softmax(q@kแต/โd)@v โ one call instead of ~20 lines, and faster because the full TรT score matrix need not be stored. is_causal=True applies the triangular mask inside the kernel, so no mask tensor needs to be materialized.
c_proj projects the concatenated heads back to n_embd. Both linear layers use bias=False: fewer parameters and a small speedup, with no noticeable quality loss at this scale.
class MLP(nn.Module):
def __init__(self, n_embd, dropout=0.0):
super().__init__()
self.fc = nn.Linear(n_embd, 4 * n_embd, bias=False)
self.proj = nn.Linear(4 * n_embd, n_embd, bias=False)
self.drop = nn.Dropout(dropout)
def forward(self, x):
return self.drop(self.proj(F.gelu(self.fc(x), approximate="tanh")))
class Block(nn.Module):
def __init__(self, n_embd, n_head, dropout=0.0):
super().__init__()
self.ln1 = nn.LayerNorm(n_embd)
self.attn = CausalSelfAttention(n_embd, n_head, dropout)
self.ln2 = nn.LayerNorm(n_embd)
self.mlp = MLP(n_embd, dropout)
def forward(self, x):
x = x + self.attn(self.ln1(x))
x = x + self.mlp(self.ln2(x))
return x
The MLP expands n_embd โ 4ยทn_embd, applies GELU, projects back. It's ~2ร the parameter count of attention per block, and it's where most of the model's "knowledge" lives. The approximate="tanh" form of GELU is the variant GPT-2 used โ a tanh-based formula that closely tracks the exact GELU.
Pre-LN residuals: x = x + attn(ln(x)), not x = ln(x + attn(x)). The normalization happens before each sublayer, on the residual branch. This is what GPT-2 onward uses โ it keeps a clean gradient path through the residual stream, which makes deep stacks far easier to train than the original post-LN design (you still want LR warmup โ see step 4).
Why two LayerNorms per block? One before attention, one before the MLP. Each sublayer sees a normalized input, adds its output back to the unnormalized residual stream. The stream itself is never normalized until the final ln_f.
class TinyGPT(nn.Module):
def __init__(self, vocab_size, n_layer=6, n_head=6, n_embd=384,
block_size=256, dropout=0.1):
super().__init__()
self.block_size = block_size
self.tok_emb = nn.Embedding(vocab_size, n_embd)
self.pos_emb = nn.Embedding(block_size, n_embd)
self.drop = nn.Dropout(dropout)
self.blocks = nn.ModuleList(
[Block(n_embd, n_head, dropout) for _ in range(n_layer)]
)
self.ln_f = nn.LayerNorm(n_embd)
self.head = nn.Linear(n_embd, vocab_size, bias=False)
self.head.weight = self.tok_emb.weight
self.apply(self._init_weights)
for name, p in self.named_parameters():
if name.endswith("proj.weight"):
nn.init.normal_(p, mean=0.0, std=0.02 / math.sqrt(2 * n_layer))
Two embeddings, added. tok_emb maps token ids to vectors; pos_emb maps positions 0โฆ255 to vectors. They're summed, then dropped out. Learned positional embeddings are simpler than RoPE (which modern models use) and work fine at this scale.
Weight tying: head.weight = tok_emb.weight. The input embedding and output projection share one vocab ร n_embd matrix. This saves 3M parameters here, and empirically it improves small models โ the same vector that means "cat" on the way in also scores "cat" on the way out.
Scaled init for residual projections. The c_proj and MLP proj layers are initialized with std 0.02/โ(2ยทn_layer) instead of 0.02. This is GPT-2's trick: as depth grows, the residual stream accumulates variance from every block, so the layers that write into the stream are initialized smaller to compensate. It's why deep pre-LN Transformers train stably without architectural changes.
๐
Parameter budget โ where the 13.8M goes
Token embedding (shared with the output head): 8000 ร 384 = 3.07M. Position embedding: 256 ร 384 = 0.10M.
Per block: attention 4 ร 384ยฒ = 0.59M, MLP 8 ร 384ยฒ = 1.18M, LayerNorms โ 0 โ 1.77M. Six blocks: 10.63M.
Total = 3.07 + 0.10 + 10.63 โ 13.8M (exactly 13,797,120). Shares: MLPs 51%, attention 26%, embeddings 22%. Without weight tying it would be 16.9M.
Check it: your training log's [train] params: line should read 13.80M.
The forward pass and the loss
TinyGPT.forward(idx, targets) takes a (B, T) batch of token ids, runs them through embeddings โ blocks โ final LayerNorm โ output head, and returns logits of shape (B, T, vocab_size). If targets is given, it also computes cross-entropy over the flattened (BยทT, vocab_size) logits against (BยทT,) targets โ exactly the next-token prediction objective from the earlier lessons, now in one F.cross_entropy call.
3
The data pipeline โ corpus, tokenizer, binary shards
Three scripts, four stages (below). build_corpus.py downloads 30 MB of TinyStories, adds a small seed Q&A set (repeated 50ร), trains an 8K BPE tokenizer, encodes everything to a single uint16 array, and writes train.bin / val.bin with a 98/2 split. train.py memory-maps those shards. infer.py loads the tokenizer and the best checkpoint.
stage 1 text files โ BPE tokenizer (vocab 8000) โ tokenizer.json
stage 2 text files โ encode with tokenizer โ uint16 array
stage 3 uint16 array โ 98/2 split โ bin/train.bin, bin/val.bin
stage 4 train.py โ np.memmap + fancy index โ (B, T) batches
Why memory-map instead of loading?
A 30 MB corpus is ~7M tokens โ 14 MB as uint16 โ small enough to load. But the same pattern scales to a 30 GB corpus without changing a line: np.memmap pages in only the slices get_batch touches, and the OS page cache handles repetition. This is the exact pattern nanoGPT uses, and it's why the training loop never mentions the dataset size.
class MemmapDataset:
def __init__(self, path, block_size):
self.data = np.memmap(path, dtype=np.uint16, mode="r")
self.block_size = block_size
def get_batch(self, batch_size):
max_start = len(self.data) - self.block_size - 1
ix = np.random.randint(0, max_start, batch_size)
offsets = np.arange(self.block_size)
x = self.data[ix[:, None] + offsets]
y = self.data[ix[:, None] + offsets + 1]
x = torch.from_numpy(x.astype(np.int64))
y = torch.from_numpy(y.astype(np.int64))
if DEVICE == "cuda":
x = x.pin_memory().to(DEVICE, non_blocking=True)
y = y.pin_memory().to(DEVICE, non_blocking=True)
return x, y
np.memmap maps the binary file into virtual memory. Reading a slice data[a:b] triggers a page fault that loads only that page from disk โ the rest stays on disk. For training, the OS page cache keeps recently-touched pages resident, so repeated epochs over a small corpus are effectively RAM-speed.
The vectorized gather self.data[ix[:, None] + offsets] builds the whole (B, T) batch with one fancy-index instead of a Python loop over B. ix is (B,) random start positions; ix[:, None] + offsets broadcasts to (B, T) index matrix; indexing the memmap with it materializes (B, T) of uint16. y is the same slice shifted by one โ next-token prediction.
Why uint16? Vocab is 8,000, so token ids fit in 16 bits. Two bytes per token instead of eight is a 4ร reduction in disk and memory bandwidth โ meaningful when the corpus is 30 GB.
Pinned memory + non_blocking=True: .pin_memory() puts the batch in page-locked RAM so the CPUโGPU copy can run asynchronously. (Without pinning, non_blocking=True silently does nothing.) A small win here, bigger when data loading is the bottleneck.
The Q&A seed โ why it's there
TinyStories is all narrative. The seed block is 20 Q&A pairs in the format <|bos|>Q: โฆ\nA: โฆ<|eos|> โ each pair wrapped in its own start/end tokens. build_corpus.py repeats the seed 50ร (about 1,000 examples, well under 1% of the data) so the model sees the format often enough to learn it. It won't become a Q&A system โ it effectively memorizes 20 answers โ but it learns to stay in format and to stop with <|eos|>. Story boundaries get the same treatment: TinyStories separates stories with the literal text <|endoftext|>, which the script rewrites to <|eos|><|bos|>, so stories and answers share one convention.
๐ฆ
The 98/2 split, and why validation matters
The last 2% of the token stream is held out. Training loss tells you the model is fitting; validation loss tells you whether it's learning patterns that generalize. This run makes ~13 passes over its ~7M training tokens (see step 4), so some train/val gap is normal โ but the moment val rises while train keeps falling, you're overfitting. The training loop saves the best-val checkpoint separately for exactly this reason.
4
The training loop โ AdamW, cosine, AMP, checkpoints
Lesson 4 explained why you need validation, warmup + cosine, and resumable checkpoints; here they appear in PyTorch, plus two new ingredients: mixed precision (AMP) and AdamW. The loop is about 40 lines and contains the core techniques of modern LLM training. Read it as five layers: schedule, forward/backward, optimizer, evaluation, checkpoint.
for it in range(start_iter, cfg.max_iters):
lr = get_lr(it, cfg)
for g in optimizer.param_groups:
g["lr"] = lr
x, y = train_ds.get_batch(cfg.batch_size)
with torch.amp.autocast(device_type="cuda", dtype=torch.float16,
enabled=(DEVICE == "cuda")):
_, loss = model(x, y)
scaler.scale(loss).backward()
if cfg.grad_clip > 0:
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), cfg.grad_clip)
scaler.step(optimizer)
scaler.update()
optimizer.zero_grad(set_to_none=True)
if (it + 1) % cfg.eval_interval == 0 or it == cfg.max_iters - 1:
metrics = estimate_loss(model, train_ds, val_ds, cfg)
torch.save({"model": raw_model.state_dict(),
"optimizer": optimizer.state_dict(),
"scaler": scaler.state_dict(),
"iter": it, "best_val": min(best_val, metrics["val"]),
"config": asdict(cfg)}, ckpt_path)
if metrics["val"] < best_val:
best_val = metrics["val"]
torch.save({"model": raw_model.state_dict(),
"config": asdict(cfg)}, "checkpoints/best.pt")
Warmup + cosine (get_lr): 300 steps of linear warmup from 0 to 3e-4, then cosine decay to 3e-5 over the remaining ~7,700 steps. Warmup prevents early divergence; cosine decay gives the model a long tail of small updates to settle into a good minimum.
AMP: autocast runs matmuls in FP16 and keeps losses/LayerNorms in FP32. GradScaler multiplies the loss by a dynamic factor before backward to prevent FP16 underflow, then unscales gradients before the optimizer step. On CPU both are no-ops (enabled=False) โ the code path is identical either way.
Grad clip: clip_grad_norm_(..., 1.0) rescales the gradient vector if its L2 norm exceeds 1.0. Cheap insurance against loss spikes; standard in every LLM training script.
Two checkpoints: ckpt.pt holds model + optimizer + scaler + iteration, so training resumes exactly where it stopped. best.pt holds only the weights that achieved the lowest val loss โ that's what infer.py loads.
The AdamW hyperparameters, and why each
| Setting | Value | Why |
lr | 3e-4 | A safe, conservative choice for this size (small GPTs often go up to 1e-3). Too high diverges; too low underfits in 8K steps. |
betas | (0.9, 0.95) | GPT-3's choice. ฮฒโ=0.95 adapts faster than the default 0.999 on short runs. |
weight_decay | 0.1 | Decoupled (AdamW). Applied to weight matrices (2-D) only; LayerNorm gains are exempt. |
grad_clip | 1.0 | Prevents occasional loss spikes from a bad batch. |
warmup_iters | 300 | ~4% of total. Enough to stabilize, short enough not to waste steps. |
batch_size | 48 | 48 ร 256 tokens = 12,288 tokens/step. Fits comfortably in 4โ8 GB GPU memory under FP16. |
max_iters | 8000 | 8000 ร 12,288 โ 98M tokens seen โ about 13 passes over the ~7M-token corpus. Past this, extra passes mostly memorize rather than learn. |
The LR schedule, drawn
warmup it โ [0, 300) lr = 3e-4 ยท (it+1)/300 โ linear ramp
cosine it โ [300, 8000) lr = 3e-5 + ยฝ(1+cos(ฯยทt))ยท(3e-4โ3e-5) โ t = progress 0โ1
floor it > 8000 lr = 3e-5 โ never goes to 0
Why a floor instead of zero? Decaying to 10% of peak (min_lr = 3e-5) is the common recipe (GPT-3, Chinchilla and nanoGPT all use it), so the final steps still make small useful updates. The last line of the formula is only a safeguard: a normal run stops at step 8000 and never reaches it.
๐
What the loss curve looks like
Random baseline: ln(8000) โ 8.99. Typical shape (your exact numbers will differ): the loss falls steeply over the first few hundred steps, then flattens slowly; expect roughly 2โ3 by step 8,000 (the reference value quoted in this lesson is ~2.9). Train and val should track each other early on. With ~13 passes over the data, a gap that opens later is the sign of overfitting โ if val rises while train keeps falling, stop (best.pt already holds your best weights) or add dropout.
5
Inference โ chat mode and top-k sampling
Lesson 5 covered temperature, top-k, top-p and repetition penalty in depth; this step only shows the minimal PyTorch version (temperature + top-k). infer.py loads best.pt and calls model.generate in one of two modes: story mode (--prompt "Once upon a time" โ your text is simply continued) or chat mode (run with no arguments โ each line you type is wrapped as Q: โฆ\nA:). Both begin with <|bos|>, like the training data. The generation loop is the standard one: crop context to block_size, forward, temperature-scale logits, top-k filter, sample.
@torch.no_grad()
def generate(self, idx, max_new_tokens, temperature=0.8, top_k=40, eos_id=None):
for _ in range(max_new_tokens):
idx_cond = idx[:, -self.block_size:]
logits, _ = self(idx_cond)
logits = logits[:, -1, :] / max(temperature, 1e-6)
if top_k:
v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
logits[logits < v[:, [-1]]] = -float("inf")
probs = F.softmax(logits, dim=-1)
nxt = torch.multinomial(probs, num_samples=1)
idx = torch.cat([idx, nxt], dim=1)
if eos_id is not None and (nxt == eos_id).any():
break
return idx
@torch.no_grad disables autograd for the whole loop. Generation doesn't need gradients, and building a graph over hundreds of sequential steps would waste memory for nothing.
The context crop idx[:, -self.block_size:] is what lets generation run past 256 tokens at all: the position table has only 256 rows, so the model sees just the most recent 256 tokens and forgets everything earlier. (This simple loop re-runs the whole window for every new token; real systems cache keys/values to avoid that.)
Temperature divides logits before softmax. 0.5 sharpens the distribution (more deterministic); 1.2 flattens it (more random). 0.8 is the default sweet spot for a small model.
Top-k keeps only the k highest-probability tokens and sets the rest to -inf, so softmax gives them zero probability. This stops the model from occasionally sampling a very unlikely token and derailing the sentence โ a common failure for small models.
EOS early stop: (nxt == eos_id).any() stops generation as soon as a sequence emits <|eos|> (written for batch size 1, which is how infer.py uses it). In chat mode the model has learned to end its answer with EOS, so this actually fires.
Wrapping the prompt
The model was trained on text shaped <|bos|>Q: โฆ\nA: โฆ<|eos|>. In chat mode, infer.py builds <|bos|>Q: {prompt}\nA: and lets generate continue. Because every seed answer ended with <|eos|> in training, the model learns to emit it when the answer is done, and generation stops. Story mode skips the Q:/A: wrapper, so the model just continues your text as a story.
Example output (illustrative โ sampling is random, yours will differ)
$ python infer.py --prompt "Once upon a time"
Once upon a time, there was a little girl named Lily. She loved to
play in the park with her friends. One day, she found a shiny red ball...
$ python infer.py
Q: What is Python?
A: Python is a high-level, interpreted programming language known for readability.
Q: What is the capital of Spain?
A: The capital of France is Paris. <- format learned, facts not
It's not GPT-4. Story mode gives coherent multi-sentence English โ a qualitative jump from the word fragments of lessons 3โ5. Chat mode only "knows" the 20 seed answers; ask anything else and it falls back to a story or a wrong seed answer. The format is learned, the facts are not โ which is what 30 MB of children's stories plus a tiny format seed can do.
๐งช
Things to try
Temperature sweep: run the same prompt at --temp 0.3, 0.8, 1.3. Watch coherence collapse into word salad at the top end.
Top-k ablation: set --top_k 0 (no filter) and compare. The long tail is exactly what top-k removes.
Resume test: checkpoints are written every 500 steps. Press Ctrl-C shortly after one is saved, then rerun python train.py. The log should say resumed: next step is โฆ and the loss should continue from about where it was โ not restart near 9.
Overfit deliberately: set --max_iters 40000 and watch val loss rise while train loss falls. At that length the model makes ~65 passes over 30 MB and starts memorizing โ the failure the 8K default is chosen to limit.
โ
To run it
Requires: Python 3.9+, PyTorch 2.3+ (the code uses torch.amp.GradScaler), plus numpy and tokenizers. A CUDA GPU is strongly recommended: a full CPU run takes many hours (roughly 10 h or more), so on CPU do the smoke test only.
Layout: five files in one folder โ architecture.py, tokenizer.py, build_corpus.py, train.py, infer.py (full code below). Then run:
pip install torch numpy tokenizers
python build_corpus.py # ~2 min: download + tokenize + shard
python train.py # ~15 min on a modern GPU
python infer.py --prompt "Once upon a time" # story mode
python infer.py # chat mode (type a question)
Check your run: build_corpus.py prints vocab_size = 8000 and roughly 7M train tokens; train.py prints params: 13.80M; the first logged loss is near 9 (ln 8000 = 8.99) and falls fast; val loss ends roughly in the 2โ3 range (reference โ 2.9); peak VRAM is a few GB.
โน๏ธ
What about micrograd.py?
It's not here โ this lesson replaces it entirely. PyTorch's autograd engine does everything micrograd.py did, faster and on tensors. If you want the from-scratch version, it's in lessons 1โ5. The five files below are the complete project.
architecture.py โ complete
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
class CausalSelfAttention(nn.Module):
def __init__(self, n_embd, n_head, dropout=0.0):
super().__init__()
assert n_embd % n_head == 0
self.n_head = n_head
self.head_dim = n_embd // n_head
self.c_attn = nn.Linear(n_embd, 3 * n_embd, bias=False)
self.c_proj = nn.Linear(n_embd, n_embd, bias=False)
self.attn_drop = nn.Dropout(dropout)
self.resid_drop = nn.Dropout(dropout)
def forward(self, x):
B, T, C = x.shape
q, k, v = self.c_attn(x).split(C, dim=2)
q = q.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
k = k.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
v = v.view(B, T, self.n_head, self.head_dim).transpose(1, 2)
y = F.scaled_dot_product_attention(
q, k, v,
dropout_p=self.attn_drop.p if self.training else 0.0,
is_causal=True,
)
y = y.transpose(1, 2).contiguous().view(B, T, C)
return self.resid_drop(self.c_proj(y))
class MLP(nn.Module):
def __init__(self, n_embd, dropout=0.0):
super().__init__()
self.fc = nn.Linear(n_embd, 4 * n_embd, bias=False)
self.proj = nn.Linear(4 * n_embd, n_embd, bias=False)
self.drop = nn.Dropout(dropout)
def forward(self, x):
return self.drop(self.proj(F.gelu(self.fc(x), approximate="tanh")))
class Block(nn.Module):
def __init__(self, n_embd, n_head, dropout=0.0):
super().__init__()
self.ln1 = nn.LayerNorm(n_embd)
self.attn = CausalSelfAttention(n_embd, n_head, dropout)
self.ln2 = nn.LayerNorm(n_embd)
self.mlp = MLP(n_embd, dropout)
def forward(self, x):
x = x + self.attn(self.ln1(x))
x = x + self.mlp(self.ln2(x))
return x
class TinyGPT(nn.Module):
def __init__(self, vocab_size, n_layer=6, n_head=6, n_embd=384,
block_size=256, dropout=0.1):
super().__init__()
self.block_size = block_size
self.tok_emb = nn.Embedding(vocab_size, n_embd)
self.pos_emb = nn.Embedding(block_size, n_embd)
self.drop = nn.Dropout(dropout)
self.blocks = nn.ModuleList(
[Block(n_embd, n_head, dropout) for _ in range(n_layer)]
)
self.ln_f = nn.LayerNorm(n_embd)
self.head = nn.Linear(n_embd, vocab_size, bias=False)
self.head.weight = self.tok_emb.weight
self.apply(self._init_weights)
for name, p in self.named_parameters():
if name.endswith("proj.weight"):
nn.init.normal_(p, mean=0.0, std=0.02 / math.sqrt(2 * n_layer))
@staticmethod
def _init_weights(m):
if isinstance(m, nn.Linear):
nn.init.normal_(m.weight, mean=0.0, std=0.02)
if m.bias is not None:
nn.init.zeros_(m.bias)
elif isinstance(m, nn.Embedding):
nn.init.normal_(m.weight, mean=0.0, std=0.02)
def forward(self, idx, targets=None):
B, T = idx.shape
assert T <= self.block_size
pos = torch.arange(T, device=idx.device)
x = self.drop(self.tok_emb(idx) + self.pos_emb(pos))
for blk in self.blocks:
x = blk(x)
x = self.ln_f(x)
logits = self.head(x)
loss = None
if targets is not None:
loss = F.cross_entropy(
logits.view(-1, logits.size(-1)), targets.view(-1), ignore_index=-1
)
return logits, loss
@torch.no_grad()
def generate(self, idx, max_new_tokens, temperature=0.8, top_k=40,
eos_id=None):
for _ in range(max_new_tokens):
idx_cond = idx[:, -self.block_size:]
logits, _ = self(idx_cond)
logits = logits[:, -1, :] / max(temperature, 1e-6)
if top_k:
v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
logits[logits < v[:, [-1]]] = -float("inf")
probs = F.softmax(logits, dim=-1)
nxt = torch.multinomial(probs, num_samples=1)
idx = torch.cat([idx, nxt], dim=1)
if eos_id is not None and (nxt == eos_id).any():
break
return idx
tokenizer.py โ complete
"""
Byte-level BPE tokenizer.
"""
import os
try:
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import ByteLevel
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
except ImportError as e:
raise SystemExit("Missing dependency - run: pip install tokenizers") from e
class BPETokenizer:
def __init__(self, tok: "Tokenizer"):
self.tok = tok
self.pad_id = tok.token_to_id("<|pad|>")
self.bos_id = tok.token_to_id("<|bos|>")
self.eos_id = tok.token_to_id("<|eos|>")
self.vocab_size = tok.get_vocab_size()
def encode(self, text, add_special=True):
ids = self.tok.encode(text).ids
if add_special:
ids = [self.bos_id] + ids + [self.eos_id]
return ids
def decode(self, ids):
return self.tok.decode(ids, skip_special_tokens=True)
def save(self, path):
self.tok.save(path)
@classmethod
def load(cls, path):
return cls(Tokenizer.from_file(path))
def train_tokenizer(text_files, vocab_size=8000, out_path="tokenizer.json"):
tok = Tokenizer(BPE(unk_token=None))
tok.pre_tokenizer = ByteLevel(add_prefix_space=False)
tok.decoder = ByteLevelDecoder()
trainer = BpeTrainer(
vocab_size=vocab_size,
special_tokens=["<|pad|>", "<|bos|>", "<|eos|>"],
initial_alphabet=ByteLevel.alphabet(),
show_progress=True,
)
tok.train(text_files, trainer)
tok.save(out_path)
return BPETokenizer(tok)
def load_tokenizer(path="tokenizer.json"):
if not os.path.exists(path):
raise FileNotFoundError(f"{path} not found โ run build_corpus.py first")
return BPETokenizer.load(path)
build_corpus.py โ complete
"""
Build corpus -> train tokenizer -> write train.bin/val.bin
"""
import os
import sys
import glob
import re
import urllib.request
import numpy as np
from tokenizer import train_tokenizer, BPETokenizer
DATA_DIR = "data"
TOK_PATH = "tokenizer.json"
OUT_DIR = "bin"
SEED_REPEAT = 50
SEED_QA = """\
Q: What is the capital of France?
A: The capital of France is Paris.
Q: What is 2 + 2?
A: 2 + 2 equals 4.
Q: What color is the sky on a clear day?
A: On a clear day the sky is blue.
Q: Who wrote Romeo and Juliet?
A: William Shakespeare wrote Romeo and Juliet.
Q: What is the boiling point of water at sea level?
A: Water boils at 100 degrees Celsius (212 Fahrenheit) at sea level.
Q: What does CPU stand for?
A: CPU stands for Central Processing Unit.
Q: What does GPU stand for?
A: GPU stands for Graphics Processing Unit.
Q: What is Python?
A: Python is a high-level, interpreted programming language known for readability.
Q: What is machine learning?
A: Machine learning is a field of AI where models learn patterns from data.
Q: What is a neural network?
A: A neural network is a set of layers of weighted connections trained to map inputs to outputs.
Q: How many days are in a week?
A: There are seven days in a week.
Q: How many hours are in a day?
A: There are 24 hours in a day.
Q: What is the largest planet in our solar system?
A: Jupiter is the largest planet in our solar system.
Q: What is the smallest prime number?
A: The smallest prime number is 2.
Q: What is the square root of 16?
A: The square root of 16 is 4.
Q: What language is used to write this program?
A: This program is written in Python.
Q: What is 10 multiplied by 5?
A: 10 multiplied by 5 equals 50.
Q: Name a primary color.
A: Red, blue, and yellow are primary colors.
Q: What is the freezing point of water?
A: Water freezes at 0 degrees Celsius (32 Fahrenheit).
Q: What is the speed of light?
A: Light travels at about 299,792,458 meters per second in a vacuum.
"""
def download_tinystories(max_mb=30):
"""Grab a chunk of TinyStories (small, clean, fast to fit)."""
dst = os.path.join(DATA_DIR, "tinystories.txt")
if os.path.exists(dst) and os.path.getsize(dst) > 1_000_000:
return dst
url = "https://huggingface.co/datasets/roneneldan/TinyStories/resolve/main/TinyStoriesV2-GPT4-train.txt"
print(f"[corpus] downloading TinyStories (~{max_mb} MB)...")
try:
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=120) as r, open(dst, "wb") as f:
got = 0
limit = max_mb * 1024 * 1024
while got < limit:
chunk = r.read(1024 * 256)
if not chunk:
break
f.write(chunk)
got += len(chunk)
print(f"[corpus] saved {got/1e6:.1f} MB -> {dst}")
return dst
except Exception as e:
print(f"[corpus] download failed ({e}); using seed only.")
return None
def format_qa(block):
"""Wrap Q&A into training format."""
pairs = re.split(r"\n(?=Q: )", block.strip())
return "\n".join("<|bos|>" + p.strip() + "<|eos|>" for p in pairs)
def main():
os.makedirs(DATA_DIR, exist_ok=True)
os.makedirs(OUT_DIR, exist_ok=True)
qa_path = os.path.join(DATA_DIR, "qa.txt")
with open(qa_path, "w", encoding="utf-8") as f:
f.write("\n".join([format_qa(SEED_QA)] * SEED_REPEAT))
print(f"[corpus] wrote seed Q&A -> {qa_path}")
story_path = download_tinystories(max_mb=30)
sources = [qa_path] + ([story_path] if story_path else [])
if not os.path.exists(TOK_PATH):
print(f"[corpus] training tokenizer (vocab 8000)...")
train_tokenizer(sources, vocab_size=8000, out_path=TOK_PATH)
print(f"[corpus] tokenizer saved -> {TOK_PATH}")
tok = BPETokenizer.load(TOK_PATH)
print(f"[corpus] tokenizer vocab_size = {tok.vocab_size}")
all_ids = []
for p in sources:
print(f"[corpus] encoding {p} ...")
with open(p, "r", encoding="utf-8", errors="ignore") as f:
text = f.read()
text = text.replace("<|endoftext|>", "<|eos|><|bos|>")
all_ids.extend(tok.encode(text, add_special=False))
assert max(all_ids) < 65536, "vocab too large for uint16 โ use uint32"
arr = np.array(all_ids, dtype=np.uint16)
print(f"[corpus] total tokens: {len(arr):,}")
n = len(arr)
split = int(n * 0.98)
train, val = arr[:split], arr[split:]
train.tofile(os.path.join(OUT_DIR, "train.bin"))
val.tofile(os.path.join(OUT_DIR, "val.bin"))
print(f"[corpus] train tokens: {len(train):,} val tokens: {len(val):,}")
print(f"[corpus] done.")
if __name__ == "__main__":
main()
train.py โ complete
import os
import math
import time
import argparse
from dataclasses import dataclass, asdict
import numpy as np
import torch
from architecture import TinyGPT
from tokenizer import BPETokenizer
torch.manual_seed(1337)
np.random.seed(1337)
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
@dataclass
class Config:
data_dir: str = "bin"
ckpt_dir: str = "checkpoints"
tok_path: str = "tokenizer.json"
block_size: int = 256
batch_size: int = 48
n_layer: int = 6
n_head: int = 6
n_embd: int = 384
dropout: float = 0.1
lr: float = 3e-4
min_lr: float = 3e-5
weight_decay: float = 0.1
beta1: float = 0.9
beta2: float = 0.95
warmup_iters: int = 300
max_iters: int = 8000
eval_interval: int = 500
eval_iters: int = 100
log_interval: int = 20
grad_clip: float = 1.0
use_compile: bool = False
vocab_size: int = 0
class MemmapDataset:
def __init__(self, path, block_size):
if not os.path.exists(path):
raise SystemExit(
f"[train] {path} not found.\n"
f" Run `python build_corpus.py` from "
f"{os.path.dirname(os.path.abspath(__file__))} first."
)
self.data = np.memmap(path, dtype=np.uint16, mode="r")
self.block_size = block_size
def get_batch(self, batch_size):
max_start = len(self.data) - self.block_size - 1
assert max_start >= 1, f"corpus too small ({len(self.data)} tokens)"
ix = np.random.randint(0, max_start, batch_size)
offsets = np.arange(self.block_size)
x = self.data[ix[:, None] + offsets]
y = self.data[ix[:, None] + offsets + 1]
x = torch.from_numpy(x.astype(np.int64))
y = torch.from_numpy(y.astype(np.int64))
if DEVICE == "cuda":
x = x.pin_memory().to(DEVICE, non_blocking=True)
y = y.pin_memory().to(DEVICE, non_blocking=True)
return x, y
def get_lr(it, cfg):
if it < cfg.warmup_iters:
return cfg.lr * (it + 1) / cfg.warmup_iters
if it > cfg.max_iters:
return cfg.min_lr
ratio = (it - cfg.warmup_iters) / (cfg.max_iters - cfg.warmup_iters)
coeff = 0.5 * (1.0 + math.cos(math.pi * ratio))
return cfg.min_lr + coeff * (cfg.lr - cfg.min_lr)
@torch.no_grad()
def estimate_loss(model, train_ds, val_ds, cfg):
model.eval()
out = {}
for name, ds in [("train", train_ds), ("val", val_ds)]:
losses = torch.zeros(cfg.eval_iters)
for k in range(cfg.eval_iters):
x, y = ds.get_batch(cfg.batch_size)
with torch.amp.autocast(device_type="cuda", dtype=torch.float16,
enabled=(DEVICE == "cuda")):
_, loss = model(x, y)
losses[k] = loss.item()
out[name] = losses.mean().item()
model.train()
return out
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--max_iters", type=int, default=None)
ap.add_argument("--batch_size", type=int, default=None)
ap.add_argument("--block_size", type=int, default=None)
ap.add_argument("--n_layer", type=int, default=None)
ap.add_argument("--n_embd", type=int, default=None)
ap.add_argument("--n_head", type=int, default=None)
ap.add_argument("--compile", action="store_true",
help="enable torch.compile (requires triton on Windows+CUDA)")
args = ap.parse_args()
cfg = Config()
if args.max_iters is not None: cfg.max_iters = args.max_iters
if args.batch_size is not None: cfg.batch_size = args.batch_size
if args.block_size is not None: cfg.block_size = args.block_size
if args.n_layer is not None: cfg.n_layer = args.n_layer
if args.n_embd is not None: cfg.n_embd = args.n_embd
if args.n_head is not None: cfg.n_head = args.n_head
if args.compile: cfg.use_compile = True
os.makedirs(cfg.ckpt_dir, exist_ok=True)
tok = BPETokenizer.load(cfg.tok_path)
cfg.vocab_size = tok.vocab_size
print(f"[train] device={DEVICE} vocab={cfg.vocab_size}")
train_ds = MemmapDataset(os.path.join(cfg.data_dir, "train.bin"), cfg.block_size)
val_ds = MemmapDataset(os.path.join(cfg.data_dir, "val.bin"), cfg.block_size)
model = TinyGPT(
vocab_size=cfg.vocab_size,
n_layer=cfg.n_layer,
n_head=cfg.n_head,
n_embd=cfg.n_embd,
block_size=cfg.block_size,
dropout=cfg.dropout,
).to(DEVICE)
n_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"[train] params: {n_params/1e6:.2f}M")
raw_model = model
if cfg.use_compile and hasattr(torch, "compile"):
model = torch.compile(model)
print("[train] torch.compile enabled")
decay = [p for p in raw_model.parameters() if p.dim() >= 2]
no_decay = [p for p in raw_model.parameters() if p.dim() < 2]
optimizer = torch.optim.AdamW(
[{"params": decay, "weight_decay": cfg.weight_decay},
{"params": no_decay, "weight_decay": 0.0}],
lr=cfg.lr, betas=(cfg.beta1, cfg.beta2),
)
scaler = torch.amp.GradScaler(device="cuda", enabled=(DEVICE == "cuda"))
start_iter = 0
best_val = float("inf")
ckpt_path = os.path.join(cfg.ckpt_dir, "ckpt.pt")
if os.path.exists(ckpt_path):
ck = torch.load(ckpt_path, map_location=DEVICE, weights_only=True)
try:
raw_model.load_state_dict(ck["model"])
except RuntimeError as e:
raise SystemExit(f"[train] {ckpt_path} doesn't match this config ({e}).\n"
f" Delete the checkpoints/ folder to start fresh.")
optimizer.load_state_dict(ck["optimizer"])
scaler.load_state_dict(ck["scaler"])
start_iter = ck["iter"] + 1
best_val = ck.get("best_val", best_val)
if start_iter >= cfg.max_iters:
raise SystemExit("[train] this run already finished. Delete checkpoints/ to retrain.")
print(f"[train] resumed: next step is {start_iter + 1} of {cfg.max_iters}")
model.train()
t0 = time.time()
running = None
for it in range(start_iter, cfg.max_iters):
lr = get_lr(it, cfg)
for g in optimizer.param_groups:
g["lr"] = lr
x, y = train_ds.get_batch(cfg.batch_size)
with torch.amp.autocast(device_type="cuda", dtype=torch.float16,
enabled=(DEVICE == "cuda")):
_, loss = model(x, y)
scaler.scale(loss).backward()
if cfg.grad_clip > 0:
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), cfg.grad_clip)
scaler.step(optimizer)
scaler.update()
optimizer.zero_grad(set_to_none=True)
running = loss.item() if running is None else 0.95 * running + 0.05 * loss.item()
if (it + 1) % cfg.log_interval == 0:
dt = max(time.time() - t0, 1e-9)
tok_s = (it + 1 - start_iter) * cfg.batch_size * cfg.block_size / dt
print(f"iter {it+1:5d} | loss {running:.4f} | lr {lr:.2e} | {tok_s/1e3:.1f}k tok/s")
if (it + 1) % cfg.eval_interval == 0 or it == cfg.max_iters - 1:
metrics = estimate_loss(model, train_ds, val_ds, cfg)
print(f"[eval] iter {it+1}: train {metrics['train']:.4f} | val {metrics['val']:.4f}")
ck = {
"model": raw_model.state_dict(),
"optimizer": optimizer.state_dict(),
"scaler": scaler.state_dict(),
"iter": it,
"best_val": min(best_val, metrics["val"]),
"config": asdict(cfg),
}
torch.save(ck, ckpt_path)
if metrics["val"] < best_val:
best_val = metrics["val"]
torch.save({"model": raw_model.state_dict(), "config": asdict(cfg)},
os.path.join(cfg.ckpt_dir, "best.pt"))
print(f"[train] checkpoint saved -> {ckpt_path}")
print("[train] done.")
if __name__ == "__main__":
main()
infer.py โ complete
import argparse
import os
import torch
from architecture import TinyGPT
from tokenizer import BPETokenizer
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
CKPT = "checkpoints/best.pt"
TOK = "tokenizer.json"
def load_model():
tok = BPETokenizer.load(TOK)
ck = torch.load(CKPT, map_location=DEVICE, weights_only=True)
cfg = ck["config"]
model = TinyGPT(
vocab_size=tok.vocab_size,
n_layer=cfg["n_layer"],
n_head=cfg["n_head"],
n_embd=cfg["n_embd"],
block_size=cfg["block_size"],
dropout=0.0,
).to(DEVICE)
model.load_state_dict(ck["model"])
model.eval()
return model, tok
def generate(model, tok, prompt, max_new=200, temperature=0.8, top_k=40):
full = f"<|bos|>{prompt.strip()}"
ids = tok.encode(full, add_special=False)
if not ids or ids[0] != tok.bos_id:
ids = [tok.bos_id] + ids
x = torch.tensor(ids, dtype=torch.long, device=DEVICE).unsqueeze(0)
with torch.amp.autocast(device_type="cuda", dtype=torch.float16, enabled=(DEVICE == "cuda")):
out = model.generate(
x, max_new_tokens=max_new,
temperature=temperature, top_k=top_k, eos_id=tok.eos_id,
)
text = tok.decode(out[0].tolist())
return text
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--prompt", type=str, default=None,
help="story mode: continue this text. Omit for chat mode.")
ap.add_argument("--temp", type=float, default=0.8)
ap.add_argument("--top_k", type=int, default=40)
ap.add_argument("--max_new", type=int, default=200)
args = ap.parse_args()
model, tok = load_model()
print(f"[infer] loaded ({DEVICE})")
if args.prompt:
print(generate(model, tok, args.prompt, args.max_new, args.temp, args.top_k))
return
print("Chat mode. Type 'quit' to exit.\n")
while True:
try:
prompt = input("Q: ").strip()
except (EOFError, KeyboardInterrupt):
print()
break
if prompt.lower() in {"quit", "exit", "q"}:
break
if not prompt:
continue
wrapped = f"Q: {prompt}\nA:"
out = generate(model, tok, wrapped, args.max_new, args.temp, args.top_k)
out = out.replace("<|bos|>", "").replace("<|eos|>", "")
print(f"A:{out.split('A:', 1)[-1]}\n")
if __name__ == "__main__":
main()
๐งฐ
Troubleshooting
Smoke test first โ --max_iters 20 --batch_size 4. Everything should finish in under a minute. Then delete the checkpoints/ folder โ otherwise the real run resumes from the smoke test's weights.
CUDA out of memory โ lower --batch_size to 24 or 12. The model is small; batch is the knob.
Loss stuck at 8.99 โ LR too high or warmup too short. Check get_lr(0) returns a small positive number.
Tokenizer mismatch on resume โ tokenizer.json was regenerated between runs. Don't delete it while training.
torch.compile fails on Windows โ expected; it needs triton. Leave --compile off.
AttributeError: torch.amp has no GradScaler โ PyTorch is older than 2.3. Upgrade it.
"this run already finished" โ a completed ckpt.pt exists. Delete checkpoints/ to retrain from scratch.