Linear Algebra Essentials: The Math Every Model in This Series Has Been Hiding

Previously in This Series

Day 5 gave this series its statistical vocabulary. Today gives it something more direct: every LogisticRegression().fit() and .predict() call from Day 2 onward has secretly just been linear algebra, running underneath a library that never bothered to mention it. By the end of this lesson, you'll have peeled that back yourself, by hand, on a model from this series — not a toy example built to make the point look clean.

Learning Objectives

• Understand what a vector and a matrix actually are in ML terms — not abstract math objects, but a single feature row and a full dataset, respectively.

• Compute a dot product by hand and confirm it's exactly what a trained logistic regression model does to produce one prediction.

• Use the L2 norm to see that a weight vector's "size" and a KNN distance calculation, from Day 4, are the same underlying operation.

• Use matrix multiplication to score an entire dataset at once, and understand why that single fact is the reason modern ML runs on GPUs.

Why This Matters

Every linear and logistic regression call in Phase 1 and Phase 2 is dot products and matrix multiplication wearing scikit-learn's clothing. Phase 4's neural networks are the same two operations, stacked in layers and run billions of times per pass. The single biggest reason deep learning became practical when it did is that GPUs are exceptionally good at exactly one thing — matrix multiplication — and today's vocabulary is the same vocabulary that explains why that hardware fact mattered at all.

It also reaches backward usefully: Day 4's KNN distance calculation and the weight vectors inside every linear model from Day 2 onward turn out to be the same mathematical object, viewed from two different angles. You've been doing linear algebra this entire series without a name for it. Today supplies the name.

Mental Model

A vector is an ingredient list for one dish — flour, sugar, eggs, in fixed amounts. A matrix is the whole cookbook: many dishes stacked together, all measured against the same ingredient list.

• A dot product is scoring one dish by multiplying each ingredient amount by how much that ingredient matters, then adding it all up. That's it. That's also, exactly, what a trained linear model does to turn one row of features into one prediction.

• Matrix multiplication is scoring every dish in the entire cookbook against the same importance list, all at once, instead of recipe by recipe.

• A norm is a single number describing how "big" or "rich" one dish's whole ingredient list is overall — not any single ingredient, but the dish as a whole.

History in 60 Seconds

• 1844 — Hermann Grassmann, an obscure German schoolteacher largely ignored in his own lifetime, publishes Die lineale Ausdehnungslehre, laying the actual mathematical foundations of linear algebra as a discipline.

• 1857 — Arthur Cayley formally introduces the matrix as a mathematical object in its own right, the direct ancestor of every X you've seen passed into a scikit-learn .fit() call this series.

• 1888 — Giuseppe Peano gives the first formal axiomatic definition of a vector space, turning Grassmann's largely unrecognized insight into the rigorous foundation modern linear algebra still stands on.

• 1986 — Rumelhart, Hinton, and Williams's backpropagation paper makes training multi-layer neural networks practical — an algorithm that is, underneath, almost entirely matrix calculus, and a direct preview of Phase 4.

Key Terminology Table

Core Concepts

Vectors and Matrices: You've Already Been Using Them

A vector, in ML terms, is just one row of features — Titanic's pclass, sex, age, sibsp, parch, fare, and embarked_enc, all for one passenger, is a 7-number vector. A matrix is what you get when you stack many of those rows together: scikit-learn's X_train has been a matrix since the very first model this series trained on Day 2. You were never given a special "linear algebra mode" to opt into — X was always a matrix, the whole time.

The Dot Product: One Operation, Almost Every Prediction

Given two vectors of the same length, the dot product multiplies them element-by-element and adds up the results into a single number. For a trained linear model, one of those vectors is the passenger's features, and the other is the model's learned weight vector — one number per feature, capturing how much that feature pushes the prediction up or down. Add a constant bias term, and that single number is the model's raw output before any final transformation. You're about to compute this by hand, on a real fitted model from this series, and check it against what scikit-learn actually returns.

Norms: One Idea, Two Things You've Already Used

A norm collapses an entire vector into one number describing its overall size. The most common one, the L2 norm, is the square root of the sum of squared elements — which is precisely the straight-line distance formula. That means Day 4's KNN distance calculation between two passengers and the magnitude of a trained model's weight vector are the exact same operation, just applied to different vectors: one to the difference between two data points, the other to the model's learned coefficients themselves. The same norm that measures "how far apart are these two passengers" also measures "how large, overall, are this model's learned weights" — which is exactly the quantity Phase 3's regularization techniques will later learn to control directly.

Matrix Multiplication: The Same Dot Product, Done at Scale

Matrix multiplication is nothing more than running the same dot product against every row of a matrix at once. Instead of looping over 268 test passengers one at a time, computing 268 separate dot products, you do it in a single operation: the entire feature matrix multiplied by the weight vector, all at once. This is not just a coding convenience — it's the exact computation a GPU is built to accelerate, and the reason today's vocabulary is the direct prerequisite for understanding why deep learning needed different hardware to become practical at all.

Visual Explanations

How a single prediction and a whole dataset of predictions use the exact same operation, just at different scales:

flowchart TD
    A[One passenger's feature vector] -->|dot product with weight vector, + bias| B[One raw score]
    B -->|sigmoid| C[One probability]
    D[Entire test set as a matrix] -->|matrix multiplication with weight vector, + bias| E[A whole vector of raw scores]
    E -->|sigmoid, applied elementwise| F[A whole vector of probabilities]

And where the L2 norm shows up doing two different-looking jobs with the same underlying math:

Hands-On Example

We're not building a toy example for this one. We're reopening a model this series already trained — Day 3 and Day 4's Titanic logistic regression — and checking, line by line, that what scikit-learn returns matches what a handwritten dot product, sigmoid, and matrix multiplication produce, to the limits of floating-point precision.

1. Refit the familiar model on the same features used since Day 3.

2. Pull out its learned weight vector and bias, and manually compute one passenger's prediction with nothing but numpy, checking it against model.decision_function() and model.predict_proba().

3. Scale that up to the entire test set at once, with a single matrix multiplication, and check it against every probability scikit-learn returns.

4. Compute an L2 norm twice — once as a distance between two passengers, once as the size of the model's weight vector — to see the same formula doing two different jobs.

Environment Setup

python -m venv venv

source venv/bin/activate
# Windows
venv\Scripts\activate

pip install numpy pandas seaborn scikit-learn

Complete Working Code

Every number in the breakdown below came from actually running this script.

"""
day_06_linear_algebra_essentials.py
Reopens this series' own Titanic logistic regression model and verifies,
by hand, that its predictions are exactly a dot product plus a sigmoid —
first for one passenger, then for the entire test set at once.
"""

import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# --- Rebuild the familiar Day 3/4 Titanic model ---
titanic = sns.load_dataset("titanic")
titanic["age"] = titanic["age"].fillna(titanic["age"].median())
titanic["embarked"] = titanic["embarked"].fillna(titanic["embarked"].mode()[0])
titanic["sex"] = titanic["sex"].map({"male": 0, "female": 1})
titanic["embarked_enc"] = titanic["embarked"].map({"S": 0, "C": 1, "Q": 2})

features = ["pclass", "sex", "age", "sibsp", "parch", "fare", "embarked_enc"]
X = titanic[features]
y = titanic["survived"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

model = LogisticRegression(max_iter=500)
model.fit(X_train, y_train)

w = model.coef_[0]    # the learned weight vector
b = model.intercept_[0]  # the learned bias

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# ============================================================
# PART 1: ONE PASSENGER, BY HAND
# ============================================================
row = X_test.iloc[0].values
manual_dot = np.dot(row, w) + b
sklearn_decision = model.decision_function(X_test.iloc[[0]])[0]

manual_prob = sigmoid(manual_dot)
sklearn_prob = model.predict_proba(X_test.iloc[[0]])[0][1]

print("=== ONE PASSENGER, BY HAND vs SKLEARN ===")
print(f"Manual dot product (row . w + b): {manual_dot:.6f}")
print(f"sklearn's decision_function():     {sklearn_decision:.6f}")
print(f"Manual sigmoid(dot product):       {manual_prob:.6f}")
print(f"sklearn's predict_proba():          {sklearn_prob:.6f}")

# ============================================================
# PART 2: THE WHOLE TEST SET, ONE MATRIX MULTIPLICATION
# ============================================================
X_test_matrix = X_test.values
manual_all_probs = sigmoid(X_test_matrix @ w + b)
sklearn_all_probs = model.predict_proba(X_test)[:, 1]

print(f"\n=== ENTIRE TEST SET, ONE MATRIX MULTIPLICATION ===")
print(f"X_test shape: {X_test_matrix.shape}  |  w shape: {w.shape}")
print(f"Max difference between manual and sklearn probabilities: "
      f"{np.max(np.abs(manual_all_probs - sklearn_all_probs)):.2e}")

# ============================================================
# PART 3: THE L2 NORM, DOING TWO DIFFERENT JOBS
# ============================================================
weight_l2_norm = np.linalg.norm(w)

p1 = X_test.iloc[0].values.astype(float)
p2 = X_test.iloc[1].values.astype(float)
manual_distance = np.sqrt(np.sum((p1 - p2) ** 2))
numpy_distance = np.linalg.norm(p1 - p2)

print(f"\n=== THE L2 NORM, TWO USES ===")
print(f"L2 norm of the model's weight vector: {weight_l2_norm:.4f}")
print(f"Manual distance between two passengers (sqrt of summed squares): {manual_distance:.4f}")
print(f"Same distance via np.linalg.norm (L2 norm of the difference):    {numpy_distance:.4f}")

Running this prints:

=== ONE PASSENGER, BY HAND vs SKLEARN ===
Manual dot product (row . w + b): -1.378810
sklearn's decision_function():     -1.378810
Manual sigmoid(dot product):       0.201200
sklearn's predict_proba():          0.201200

=== ENTIRE TEST SET, ONE MATRIX MULTIPLICATION ===
X_test shape: (268, 7)  |  w shape: (7,)
Max difference between manual and sklearn probabilities: 2.22e-16

=== THE L2 NORM, TWO USES ===
L2 norm of the model's weight vector: 2.7981
Manual distance between two passengers (sqrt of summed squares): 48.6270
Same distance via np.linalg.norm (L2 norm of the difference):    48.6270

Code Breakdown

One passenger, by hand. np.dot(row, w) + b is the entire "prediction" step of a logistic regression model, with nothing hidden: multiply each feature by its learned weight, sum the results, add the bias. That number, -1.378810, matches scikit-learn's own decision_function() exactly. Push it through the sigmoid function — which squashes any real number into a 0-to-1 probability — and you get 0.2012, matching predict_proba() exactly. There was no black box here. The library was just doing this faster.

The whole test set, one matrix multiplication. X_test_matrix @ w + b runs the identical operation against all 268 passengers simultaneously, producing a vector of 268 raw scores in one line instead of a loop. The maximum difference from scikit-learn's own batch predictions is 2.22e-16 — floating-point noise, not a real discrepancy. This is the entire reason matrix multiplication matters practically: the same math, the same answer, but one hardware-friendly operation instead of hundreds of individual ones.

The L2 norm, two uses. np.linalg.norm(w) treats the model's seven learned weights as a single vector and reports its overall size: 2.7981. np.linalg.norm(p1 - p2) treats the difference between two passengers' feature vectors as a vector and reports its size: 48.6270 — and that's just Day 4's KNN distance calculation, computed with the exact same function. Same formula, same code, two completely different-looking jobs.

Common Mistakes

• Treating "vector" and "matrix" as math-class abstractions, disconnected from X and a single row of X. They aren't abstractions — they're the exact objects this series has used since Day 2.

• Assuming matrix multiplication and elementwise multiplication are the same thing. They aren't; matrix multiplication involves dot products across rows and columns, not just multiplying corresponding entries — mixing these up is one of the most common early numpy shape-mismatch bugs.

• Forgetting that the bias term exists. A dot product alone, without the bias, will not match a real model's output — every model in this series has a learned bias sitting alongside its weights.

• Confusing a vector's norm with its length (number of elements). A norm is a measure of magnitude, not a count of how many numbers are in the vector — those are unrelated quantities that happen to share casual language.

• Assuming "linear algebra" means the model itself is simple. Today's lesson covers what's underneath logistic regression specifically; Phase 4's neural networks use the exact same operations, just stacked and repeated, not a fundamentally different kind of math.

Best Practices

• When debugging a model that isn't behaving as expected, check the shapes of every vector and matrix involved before checking anything else — shape mismatches are the single most common practical linear algebra error in ML code.

• Use @ (or np.matmul) for matrix multiplication and reserve * for elementwise operations — numpy will happily run the wrong one without complaint if the shapes happen to broadcast.

• When you want to understand what a trained linear model is actually doing, inspect its weight vector directly, the way today's code did — it's rarely more than a few lines, and it turns "the model decided" into "the model computed this specific number, because of these specific weights."

• Treat matrix multiplication as the default mental model for "doing the same thing to every row at once," instead of reaching for a loop — it's both faster and the more accurate way to think about what these models are actually doing.

Production Perspective

GPUs became the dominant hardware for ML largely as a fortunate accident of their original purpose: they were built for graphics rendering, which already meant enormous numbers of matrix multiplications running in parallel, and it turned out neural network training needed exactly that same capability — a direct consequence of today's lesson, not a separate fact to memorize. Understanding that a model's forward pass is matrix multiplication is the reason "how big is my model" and "how much compute does inference cost" are, underneath, the same kind of question: both are really asking about the size of the matrices involved.

In production, numerical precision becomes a real, practical concern in exactly the spot today's code touched on: the 2.22e-16 gap between the manual and library-computed probabilities is floating-point noise, not a bug — but at large enough scale, accumulated floating-point error across millions of matrix operations is a genuine source of subtle production bugs, especially when comparing model outputs across different hardware or different numeric precision settings (float32 vs. float64), a topic that becomes directly relevant once Phase 4 and Phase 5 introduce GPU training and deployment.

Cost-wise, this is where "model size" stops being an abstraction. A model's weight matrix size directly determines both memory footprint and the number of multiplications needed per prediction — which is exactly why larger models cost more to serve, in a way you can now calculate rather than just intuit.

Real-World Applications

• Recommendation systems, where a user's preference vector and an item's feature vector are scored against each other with a dot product, at the scale of millions of users and items.

• Search and information retrieval, where matching a query to documents often comes down to a dot product or cosine similarity between vector representations.

• Computer vision, where convolutional layers (Phase 4) are, underneath, structured matrix multiplications applied to image patches.

• Word and sentence embeddings, where similarity between two pieces of text is measured almost entirely with a dot product (often normalized into cosine similarity) between two learned vectors — the exact mechanism behind Phase 6's embeddings and vector databases.

• Any production ML serving system, where understanding that inference cost scales with matrix size directly informs real capacity-planning decisions.

Interview Questions

1. What is a dot product, and how does it relate to a trained linear model's prediction? A dot product multiplies two vectors element-by-element and sums the results into a single number. A trained linear model's raw prediction is exactly the dot product of the input features and the model's learned weights, plus a bias term.

2. What's the difference between matrix multiplication and elementwise multiplication? Elementwise multiplication multiplies corresponding entries of two same-shaped arrays independently. Matrix multiplication combines rows and columns through sums of products — it's the batched generalization of the dot product, not a simple per-entry operation.

3. Why is the L2 norm relevant to both distance calculations and model regularization? Because both are the same operation — the square root of a sum of squares — applied to different vectors: the difference between two points for distance, or a model's own weight vector for regularization, which directly penalizes large weight magnitudes.

4. Why does matrix multiplication matter for understanding why GPUs accelerate deep learning? Because a neural network's forward pass is fundamentally a sequence of matrix multiplications, and GPUs are hardware specifically optimized to perform huge numbers of these in parallel — the hardware advantage exists because of the underlying math, not despite it.

5. If a model's predictions don't match a manual recomputation using its weights, what would you check first? Whether the bias/intercept term was included, whether the feature order matches the order the weights were learned in, and whether any final transformation (like a sigmoid) was applied — all three are common, easy-to-miss sources of mismatch.

6. What does the size (norm) of a model's weight vector tell you, practically? Roughly, how much the model is relying on large coefficient values to make predictions — a useful early signal for whether regularization might help, since very large weights are often associated with overfitting.

7. Why is it more efficient to predict on an entire dataset with one matrix multiplication than to loop over rows? A single matrix multiplication can be executed as one optimized, parallelizable operation, while a loop performs the same work serially with added per-iteration overhead — the mathematical result is identical, but the computational cost is not.

8. What's a common bug caused by confusing vector shapes in matrix operations? Unintentional broadcasting — numpy silently producing a result with the wrong shape instead of raising an error, because the shapes happened to be compatible in a way the programmer didn't intend, often producing wrong but plausible-looking numbers rather than a clear failure.

Self-Assessment

• Can you compute, by hand, what a logistic regression model would predict for a feature vector you make up yourself, given its weights and bias?

• Could you explain why Day 4's KNN distance and a model's weight-vector norm are the same underlying formula, to someone who's never heard the word "norm"?

• Can you explain, without looking back at the code, why matrix multiplication and elementwise multiplication give different results for the same two arrays?

• Can you describe, in plain language, why GPUs matter for deep learning, tracing the explanation back to matrix multiplication specifically?

Portfolio Challenges

Beginner Challenge

Pick any two passengers from the Titanic test set and compute their L2 distance by hand, then verify it with np.linalg.norm. Repeat with a different pair and compare which two passengers are "closer" in feature space — and whether that matches your intuition about how similar they seem.

Intermediate Challenge

Refit today's logistic regression model using only three features instead of seven. Recompute the manual dot product for one passenger, confirm it still matches decision_function(), and report how the weight vector's L2 norm changes compared to the seven-feature version.

Advanced Challenge

Implement matrix multiplication from scratch using nested loops (no @, no np.dot, no np.matmul), confirm it produces the same result as X_test_matrix @ w on a small slice of the test set, then time both versions on the full test set and report the speed difference.

Advanced Insights

Phase 4's neural networks are today's lesson, repeated: each layer is a matrix multiplication followed by a nonlinear function, just like the sigmoid in today's logistic regression, stacked multiple times instead of once. Understanding today's single-layer version by hand is the entire prerequisite for understanding a deep network as "the same thing, more times," rather than as a fundamentally different kind of object.

It's also worth knowing about eigenvectors and eigenvalues, which this lesson deliberately left out to stay at the essentials level: they're the mathematical machinery behind Principal Component Analysis, which Phase 3 will use for dimensionality reduction, and they describe the directions a matrix stretches space along without rotating it. You don't need them yet — but when PCA arrives, today's comfort with vectors and matrices is what will make eigenvectors feel like a natural next step instead of an entirely new subject.

Key Takeaways

• A vector is one feature row; a matrix is a full dataset. This series has used both since Day 2, without naming them.

• A dot product is the entire computation behind a linear model's raw prediction — verified today to match scikit-learn's decision_function() exactly, to floating-point precision.

• The L2 norm is one formula doing two jobs already seen in this series: Day 4's KNN distance, and a model's weight-vector magnitude, which previews Phase 3's regularization.

• Matrix multiplication is the same dot product, batched across an entire dataset at once — confirmed today against all 268 test-set predictions simultaneously — and it's the specific operation that makes GPUs relevant to machine learning at all.

What's Next

Day 7 closes out Phase 1 with Data Preprocessing — scaling, encoding, and preparing raw features properly, building directly on today's lesson, since most preprocessing decisions exist specifically to keep dot products and distance calculations from being distorted by mismatched feature scales.

References

Beginner

• Google: Machine Learning Crash Course — includes a module on the linear algebra foundations used throughout ML.

• NumPy: Linear algebra documentation — official reference for the dot product, matrix multiplication, and norm functions used in this article's code.

Intermediate

• Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani, An Introduction to Statistical Learning — free at statlearning.com, with linear regression presented explicitly in matrix form.

Professional

• Aurélien Géron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (O'Reilly) — covers the vectorized, matrix-based implementation of the models used throughout this series. Available through O'Reilly and major booksellers.

Advanced

• Gilbert Strang, Introduction to Linear Algebra (Wellesley-Cambridge Press) — the standard reference for the full mathematical treatment behind today's essentials-level introduction, including the eigenvectors mentioned in Advanced Insights.