Interactive Lesson Β· Start Here

AI from Zero

Understand what artificial intelligence actually is by building the smallest possible model from scratch β€” a line with just two parameters.

What it is

A tiny model

A line Ε· = aΒ·x + b with only 2 trainable parameters. It uses the same training loop as every modern AI model, stripped to its core.

What it does

Learns from data

We feed it (x, y) pairs from the rule y = 3x + 1. Gradient descent discovers a β‰ˆ 3 and b β‰ˆ 1 on its own.

What you get

A working model

A frozen line that answers new inputs. Give it x = 7, it returns Ε· = 22.011 β€” without ever seeing the answer.

Ε· = aΒ·x + b
Start the lesson ↓

The Core Idea in One Sentence

AI = regression + self-learned features + scale

Every AI model β€” from a 2-parameter line to a large language model β€” follows this same formula.

This lesson breaks that formula down into its three parts, using the simplest possible model as a foundation. By the end, you'll see that the training loop behind y = ax + b is the same loop behind every modern AI model β€” only the architecture, the data, and the scale change.

Start with the Smallest Model

Before looking at large models, understand this:

Ε· = aΒ·x + b

a = slope

How steep the line is. A single number.

b = intercept

Where the line crosses zero. A single number.

x = input

Given by the user at inference time.

Ε· = output

Computed by the model from the input.

This model has exactly 2 trainable parameters: a and b. That's the entire "brain" β€” two numbers.

What is NOT a parameter

ThingRoleTrained?
xInput data❌ No
yCorrect answer (training data)❌ No
a, bModel parametersβœ… Yes
Rule of thumb

Parameters = what the model learns. Data = what it learns from. They are never the same thing.

The 7-Step Loop (How Every AI Works)

This loop does not change β€” not for y = ax + b, not for a large language model.

  1. Define the architecture (the formula) We write Ε· = aΒ·x + b in code. This is fixed β€” it never changes during training. It defines the shape of what the model can learn.
  2. Start parameters random a and b begin at random values (e.g., a = 0.5, b = -0.2). At this point, the model is garbage β€” it knows nothing. We do NOT know a = 3, b = 1 yet. That is what training will discover.
  3. Train on data We feed (x, y) pairs. Gradient descent nudges a and b to reduce the loss. The loss here is the sum of squared errors:
    loss = Ξ£(aΒ·x + b βˆ’ y)Β²

    Ξ£ means "add up over all training examples." So this formula says: for each example, compute the error, square it, and add them all up. (In practice, most code divides by the number of examples β€” that's the "mean" in Mean Squared Error.)

    Example training data (from the rule y = 3x + 1):
    x=1 β†’ y=4
    x=2 β†’ y=7
    x=3 β†’ y=10
  4. Parameters converge After enough steps, a β‰ˆ 3 and b β‰ˆ 1. The parameters converged to the rule the data encodes. Different data β†’ different a, b.
  5. Save only the numbers We save only {a, b} to disk (e.g., model.json). The formula itself is not saved β€” it lives in code. That file is "the model."
    { "a": 3.002, "b": 0.997 }
  6. Load at inference (frozen) At inference, we load a, b back into the same formula. The parameters are frozen β€” no more training.
  7. User gives x β†’ model returns Ε· The user supplies only x. The model computes Ε· = aΒ·x + b and returns it. The user never gives y β€” producing it is the model's job.
Try it live β€” the inference step

The model has been trained: a β‰ˆ 3.002, b β‰ˆ 0.997. Adjust the input x and see the output update.

Ε· = 3.002 Γ— 7 + 0.997 = 22.011

Why a = 3, b = 1? (A Common Confusion)

We do not know a = 3, b = 1 before training. We only know it after training β€” because the data came from 3x + 1.

StageDo we know a, b?
Before training❌ No β€” random init
During trainingπŸ”„ Being adjusted
After trainingβœ… Converged to β‰ˆ3, β‰ˆ1
Why this matters

If we hardcoded 3 and 1, we would not have AI β€” we would have a calculator with the answer key taped on. Training = discovering the unknowns from data.

Training vs. Inference (Two Separate Phases)

TrainingInference
WeightsBeing updatedFrozen
User involved?NoYes
y present?Yes (correct answer)No (model produces it)
What happensLearn a, b from dataCompute Ε· from x
Key insight

The model does not update itself while you use it. Learning and using are separate phases.

The Model's Limits

A model trained on one line (3x+1) can only answer that line. It cannot do multiplication, language, or anything else.

The model's "world"

Exactly the patterns in its training data, filtered by its architecture.

To do more, you need:

More data

More examples for the model to learn from.

A richer architecture

Not just a line β€” something that can express more complex patterns.

Architecture vs. Internal Representations

These are two different things β€” do not confuse them.

ArchitectureInternal Representations
WhatThe fixed formula/layoutLearned signals flowing inside
Who decidesHuman (code)Model (during training)
Changes in training?❌ Noβœ… Yes
Example (y=ax+b)Ε· = aΒ·x + bThe learned a, b
Example (large model)Transformer layoutHidden vectors per word
Analogy

Architecture = the factory layout (fixed). Internal representations = what is on the conveyor belt at each station (transformed, learned). In a language model, the vector for "bank" shifts toward "riverbank" when it appears near "river" β€” that shift is the learned representation.

Is architecture mandatory?

Yes. No architecture = no parameters = no training. You can borrow an architecture (use a Transformer), search for one (Neural Architecture Search), but you cannot train without one.

The Full Formula

AI = regression + self-learned features + scale

Part 1: Regression (the engine)

Every AI model minimizes a loss over data by adjusting parameters. That is regression:

  • y = ax + b β†’ minimize MSE (Mean Squared Error) β†’ linear regression
  • A language model β†’ minimize cross-entropy β†’ regression on a probability distribution (i.e., the model predicts the probability of each possible next word)
Same role, different formula

MSE and cross-entropy are not the same math. They play the same role β€” a loss minimized by gradient descent β€” but their formulas differ because the output type differs (a single number vs. a probability distribution).

Part 2: Self-Learned Features

Classical regressionAI
Human picks x columns (e.g., size, bedrooms)Model gets raw data (pixels, text)
Human decides what mattersModel discovers what matters

"Self-learned features" means: the model learns by itself which internal signals to extract from raw data β€” instead of a human hand-picking them.

Important

Self-learned features happen during training, baked into weights. They have nothing to do with user input at inference.

Part 3: Scale (the multiplier)

Scale = parameters Γ— data Γ— compute

Dimensiony=ax+bLarge language model (reported estimates)
Parameters2hundreds of billions to trillions
Data3 pointstrillions of tokens
Computea calculatorthousands of GPUs

Why scale matters: At sufficient scale, new behaviors emerge that small models never show β€” reasoning, translation, code generation. Nobody programmed these in; they emerged.

Live Emulation: Train, Then Test

The two Python scripts below implement the 7-step loop. You don't need Python installed β€” each code block has a Run button that executes the same logic directly in your browser. Run train.py first to train the model, then run infer.py to test it with your own input.

How the cache works

In real Python, train.py writes {a, b} to ./cache/model.json, and infer.py reads it back. In this emulator, the "cache" is kept in browser memory β€” but the separation is identical: training produces the parameters, inference only consumes them.

train.py
Python 3
# train.py β€” learn y = a*x + b from data, save params to cache
import json
import os
import random

# ---------- Step 1: define the architecture (the formula) ----------
# The model is simply: y_hat = a * x + b
# The architecture is fixed. Only a and b will change.

# ---------- Step 2: start parameters random ----------
a = random.uniform(-1, 1)
b = random.uniform(-1, 1)
print(f"start: a={a:.4f}, b={b:.4f}")

# ---------- Step 3: training data + gradient descent ----------
# Data comes from the hidden rule y = 3x + 1
data = [(1, 4), (2, 7), (3, 10)]

lr = 0.01        # learning rate: how big each step is
epochs = 2000    # how many passes over the data

for epoch in range(epochs):
    grad_a = 0.0
    grad_b = 0.0

    for x, y in data:
        y_hat = a * x + b
        error = y_hat - y
        # gradient of squared error w.r.t. a and b
        grad_a += 2 * error * x
        grad_b += 2 * error

    # average over the dataset (this is the "mean" in MSE)
    grad_a /= len(data)
    grad_b /= len(data)

    # update both parameters in the opposite direction of the gradient
    a -= lr * grad_a
    b -= lr * grad_b

    if (epoch + 1) % 500 == 0:
        # loss = mean squared error, just for logging
        loss = sum((a * x + b - y) ** 2 for x, y in data) / len(data)
        print(f"epoch {epoch+1:4d}: a={a:.4f}, b={b:.4f}, loss={loss:.6f}")

# ---------- Step 4: converged ----------
print(f"final:  a={a:.4f}, b={b:.4f}")

# ---------- Step 5: save ONLY the numbers to cache ----------
os.makedirs("cache", exist_ok=True)
with open("cache/model.json", "w") as f:
    json.dump({"a": a, "b": b}, f, indent=2)

print("model saved to cache/model.json")
infer.py
Python 3
# infer.py β€” load the cached model and answer new inputs
import json

# ---------- Step 6: load the frozen parameters ----------
with open("cache/model.json") as f:
    model = json.load(f)

a = model["a"]
b = model["b"]
print(f"loaded model: a={a:.4f}, b={b:.4f}")

# ---------- Step 7: user gives x, model returns y_hat ----------
while True:
    raw = input("x (or 'q' to quit): ")
    if raw.lower() == "q":
        break
    try:
        x = float(raw)
    except ValueError:
        print("  please enter a number")
        continue

    y_hat = a * x + b
    print(f"  y_hat = {a:.4f} * {x} + {b:.4f} = {y_hat:.4f}")
The whole loop, split across two files

train.py does steps 1–5 (define, random init, train, converge, save). infer.py does steps 6–7 (load, infer). The cache file is the only bridge between them β€” and the Run buttons above follow that exact separation: training must finish before inference becomes available.

Common Misconceptions

"AI updates itself while I talk to it"
βœ“ No β€” weights are frozen at inference
"Self-learned means self-updating"
βœ“ No β€” features are learned during training only
"Vector vs. number separates AI from regression"
βœ“ No β€” regression can output vectors; AI can output numbers
"AI is just regression"
⚠ Half-true β€” it is regression + features + scale
"Regression includes self-learned features"
βœ“ No β€” those are the AI add-ons
"We know a=3, b=1 in advance"
βœ“ No β€” training discovers them

The Ladder: From Smallest to Largest

Same 7-step loop. Same regression engine. Different architecture + data + scale.

y = ax + b
2 params
1 sigmoid neuron
~4 params
UltraTinyLLM
732 params
MiniMind
26M params
Large language model
hundreds of billions+

UltraTinyLLM and MiniMind are small open-source teaching models. They are listed here only to show scale β€” not to imply they are required for this lesson.

The point

The 7-step loop does not change. The engine does not change. Only the architecture, the data, and the scale grow.

Knowledge Check

Answer these 5 questions to check your understanding. You can retake the quiz as many times as you like.

Q1. In the model Ε· = aΒ·x + b, which values are the trainable parameters?

Q2. What is the correct order of the 7-step loop?

Q3. At inference time, what does the user provide?

Q4. What does "self-learned features" mean?

Q5. Which statement best captures the relationship between AI and regression?

0 / 5

Final Takeaway

The whole lesson in two lines

  1. AI is regression β€” the same engine that fits y = ax + b also trains language models.
  2. What makes it "AI" is that the model learns its own features, and it is big enough that new capabilities emerge.

The math never changed from y = ax + b to a large language model. What changed is:

  1. Who picks the features (human β†’ model)
  2. How big the thing is (2 params β†’ hundreds of billions+)
Try it yourself

Save the two Python files above as train.py and infer.py in the same folder, run python train.py, then python infer.py. That's the entire loop β€” from random numbers to a working model β€” in under 60 lines of code. Or just use the live emulator in Section 9.

You've reached the end.

You started with two numbers and arrived at the full picture. The core concept is the same at every level.

AI = regression + features + scale