Every AI model β from a 2-parameter line to a large language model β follows this same formula.
AI from Zero
Understand what artificial intelligence actually is by building the smallest possible model from scratch β a line with just two parameters.
A tiny model
A line Ε· = aΒ·x + b with only 2 trainable parameters. It uses the same training loop as every modern AI model, stripped to its core.
Learns from data
We feed it (x, y) pairs from the rule y = 3x + 1. Gradient descent discovers a β 3 and b β 1 on its own.
A working model
A frozen line that answers new inputs. Give it x = 7, it returns Ε· = 22.011 β without ever seeing the answer.
The Core Idea in One Sentence
This lesson breaks that formula down into its three parts, using the simplest possible model as a foundation. By the end, you'll see that the training loop behind y = ax + b is the same loop behind every modern AI model β only the architecture, the data, and the scale change.
Start with the Smallest Model
Before looking at large models, understand this:
a = slope
How steep the line is. A single number.
b = intercept
Where the line crosses zero. A single number.
x = input
Given by the user at inference time.
Ε· = output
Computed by the model from the input.
This model has exactly 2 trainable parameters: a and b. That's the entire "brain" β two numbers.
What is NOT a parameter
| Thing | Role | Trained? |
|---|---|---|
x | Input data | β No |
y | Correct answer (training data) | β No |
a, b | Model parameters | β Yes |
Parameters = what the model learns. Data = what it learns from. They are never the same thing.
The 7-Step Loop (How Every AI Works)
This loop does not change β not for y = ax + b, not for a large language model.
-
Define the architecture (the formula)
We write
Ε· = aΒ·x + bin code. This is fixed β it never changes during training. It defines the shape of what the model can learn. -
Start parameters random
aandbbegin at random values (e.g.,a = 0.5,b = -0.2). At this point, the model is garbage β it knows nothing. We do NOT knowa = 3, b = 1yet. That is what training will discover. -
Train on data
We feed
(x, y)pairs. Gradient descent nudgesaandbto reduce the loss. The loss here is the sum of squared errors:loss = Ξ£(aΒ·x + b β y)Β²Ξ£ means "add up over all training examples." So this formula says: for each example, compute the error, square it, and add them all up. (In practice, most code divides by the number of examples β that's the "mean" in Mean Squared Error.)
Example training data (from the ruley = 3x + 1):x=1 β y=4
x=2 β y=7
x=3 β y=10 -
Parameters converge
After enough steps,
a β 3andb β 1. The parameters converged to the rule the data encodes. Different data β differenta, b. -
Save only the numbers
We save only
{a, b}to disk (e.g.,model.json). The formula itself is not saved β it lives in code. That file is "the model."{ "a": 3.002, "b": 0.997 } -
Load at inference (frozen)
At inference, we load
a, bback into the same formula. The parameters are frozen β no more training. -
User gives x β model returns Ε·
The user supplies only
x. The model computesΕ· = aΒ·x + band returns it. The user never givesyβ producing it is the model's job.
The model has been trained: a β 3.002, b β 0.997. Adjust the input x and see the output update.
Why a = 3, b = 1? (A Common Confusion)
We do not know a = 3, b = 1 before training. We only know it after training β because the data came from 3x + 1.
| Stage | Do we know a, b? |
|---|---|
| Before training | β No β random init |
| During training | π Being adjusted |
| After training | β Converged to β3, β1 |
If we hardcoded 3 and 1, we would not have AI β we would have a calculator with the answer key taped on. Training = discovering the unknowns from data.
Training vs. Inference (Two Separate Phases)
| Training | Inference | |
|---|---|---|
| Weights | Being updated | Frozen |
| User involved? | No | Yes |
y present? | Yes (correct answer) | No (model produces it) |
| What happens | Learn a, b from data | Compute Ε· from x |
The model does not update itself while you use it. Learning and using are separate phases.
The Model's Limits
A model trained on one line (3x+1) can only answer that line. It cannot do multiplication, language, or anything else.
Exactly the patterns in its training data, filtered by its architecture.
To do more, you need:
More data
More examples for the model to learn from.
A richer architecture
Not just a line β something that can express more complex patterns.
Architecture vs. Internal Representations
These are two different things β do not confuse them.
| Architecture | Internal Representations | |
|---|---|---|
| What | The fixed formula/layout | Learned signals flowing inside |
| Who decides | Human (code) | Model (during training) |
| Changes in training? | β No | β Yes |
Example (y=ax+b) | Ε· = aΒ·x + b | The learned a, b |
| Example (large model) | Transformer layout | Hidden vectors per word |
Architecture = the factory layout (fixed). Internal representations = what is on the conveyor belt at each station (transformed, learned). In a language model, the vector for "bank" shifts toward "riverbank" when it appears near "river" β that shift is the learned representation.
Is architecture mandatory?
Yes. No architecture = no parameters = no training. You can borrow an architecture (use a Transformer), search for one (Neural Architecture Search), but you cannot train without one.
The Full Formula
Part 1: Regression (the engine)
Every AI model minimizes a loss over data by adjusting parameters. That is regression:
y = ax + bβ minimize MSE (Mean Squared Error) β linear regression- A language model β minimize cross-entropy β regression on a probability distribution (i.e., the model predicts the probability of each possible next word)
MSE and cross-entropy are not the same math. They play the same role β a loss minimized by gradient descent β but their formulas differ because the output type differs (a single number vs. a probability distribution).
Part 2: Self-Learned Features
| Classical regression | AI |
|---|---|
Human picks x columns (e.g., size, bedrooms) | Model gets raw data (pixels, text) |
| Human decides what matters | Model discovers what matters |
"Self-learned features" means: the model learns by itself which internal signals to extract from raw data β instead of a human hand-picking them.
Self-learned features happen during training, baked into weights. They have nothing to do with user input at inference.
Part 3: Scale (the multiplier)
Scale = parameters Γ data Γ compute
| Dimension | y=ax+b | Large language model (reported estimates) |
|---|---|---|
| Parameters | 2 | hundreds of billions to trillions |
| Data | 3 points | trillions of tokens |
| Compute | a calculator | thousands of GPUs |
Why scale matters: At sufficient scale, new behaviors emerge that small models never show β reasoning, translation, code generation. Nobody programmed these in; they emerged.
Live Emulation: Train, Then Test
The two Python scripts below implement the 7-step loop. You don't need Python installed β each code block has a Run button that executes the same logic directly in your browser. Run train.py first to train the model, then run infer.py to test it with your own input.
In real Python, train.py writes {a, b} to ./cache/model.json, and infer.py reads it back. In this emulator, the "cache" is kept in browser memory β but the separation is identical: training produces the parameters, inference only consumes them.
# train.py β learn y = a*x + b from data, save params to cache
import json
import os
import random
# ---------- Step 1: define the architecture (the formula) ----------
# The model is simply: y_hat = a * x + b
# The architecture is fixed. Only a and b will change.
# ---------- Step 2: start parameters random ----------
a = random.uniform(-1, 1)
b = random.uniform(-1, 1)
print(f"start: a={a:.4f}, b={b:.4f}")
# ---------- Step 3: training data + gradient descent ----------
# Data comes from the hidden rule y = 3x + 1
data = [(1, 4), (2, 7), (3, 10)]
lr = 0.01 # learning rate: how big each step is
epochs = 2000 # how many passes over the data
for epoch in range(epochs):
grad_a = 0.0
grad_b = 0.0
for x, y in data:
y_hat = a * x + b
error = y_hat - y
# gradient of squared error w.r.t. a and b
grad_a += 2 * error * x
grad_b += 2 * error
# average over the dataset (this is the "mean" in MSE)
grad_a /= len(data)
grad_b /= len(data)
# update both parameters in the opposite direction of the gradient
a -= lr * grad_a
b -= lr * grad_b
if (epoch + 1) % 500 == 0:
# loss = mean squared error, just for logging
loss = sum((a * x + b - y) ** 2 for x, y in data) / len(data)
print(f"epoch {epoch+1:4d}: a={a:.4f}, b={b:.4f}, loss={loss:.6f}")
# ---------- Step 4: converged ----------
print(f"final: a={a:.4f}, b={b:.4f}")
# ---------- Step 5: save ONLY the numbers to cache ----------
os.makedirs("cache", exist_ok=True)
with open("cache/model.json", "w") as f:
json.dump({"a": a, "b": b}, f, indent=2)
print("model saved to cache/model.json")
# infer.py β load the cached model and answer new inputs
import json
# ---------- Step 6: load the frozen parameters ----------
with open("cache/model.json") as f:
model = json.load(f)
a = model["a"]
b = model["b"]
print(f"loaded model: a={a:.4f}, b={b:.4f}")
# ---------- Step 7: user gives x, model returns y_hat ----------
while True:
raw = input("x (or 'q' to quit): ")
if raw.lower() == "q":
break
try:
x = float(raw)
except ValueError:
print(" please enter a number")
continue
y_hat = a * x + b
print(f" y_hat = {a:.4f} * {x} + {b:.4f} = {y_hat:.4f}")
train.py does steps 1β5 (define, random init, train, converge, save). infer.py does steps 6β7 (load, infer). The cache file is the only bridge between them β and the Run buttons above follow that exact separation: training must finish before inference becomes available.
Common Misconceptions
The Ladder: From Smallest to Largest
Same 7-step loop. Same regression engine. Different architecture + data + scale.
UltraTinyLLM and MiniMind are small open-source teaching models. They are listed here only to show scale β not to imply they are required for this lesson.
The 7-step loop does not change. The engine does not change. Only the architecture, the data, and the scale grow.
Knowledge Check
Answer these 5 questions to check your understanding. You can retake the quiz as many times as you like.
Final Takeaway
The whole lesson in two lines
- AI is regression β the same engine that fits
y = ax + balso trains language models. - What makes it "AI" is that the model learns its own features, and it is big enough that new capabilities emerge.
The math never changed from y = ax + b to a large language model. What changed is:
- Who picks the features (human β model)
- How big the thing is (2 params β hundreds of billions+)
Save the two Python files above as train.py and infer.py in the same folder, run python train.py, then python infer.py. That's the entire loop β from random numbers to a working model β in under 60 lines of code. Or just use the live emulator in Section 9.
You've reached the end.
You started with two numbers and arrived at the full picture. The core concept is the same at every level.