Statistics Essentials for Machine Learning: Mean, Variance, Sampling, and Significance

Previously in This Series

Day 4 found a real confound — age-missingness in the Titanic dataset mostly tracked passenger class — but left one thread hanging: a residual gap in 1st class (65.6% survival vs. 46.7%, with only 30 missing-age passengers) that we explicitly said wasn't "fully ruled out." We had no proper tool to settle it; we just had to shrug and move on. Today we get that tool, and we use it on exactly that question before the lesson is over.

Learning Objectives

• Know when to trust the mean and when to reach for the median instead, and why the difference isn't just a style preference.

• Understand what a probability distribution is, and why the normal distribution specifically keeps showing up across machine learning.

• Use the Central Limit Theorem to explain, with an actual formula, why bigger samples give you more reliable estimates — and why "more data" has diminishing returns.

• Run a real hypothesis test and correctly interpret what a p-value does and doesn't tell you, including finally closing Day 4's open question.

Why This Matters

Every evaluation metric Phase 3 will teach you is a statistic, and every claim shaped like "Model A beat Model B" is implicitly a hypothesis-testing question, whether anyone phrases it that way or not. Skip this lesson and Phase 3 will feel like a list of formulas; sit with it and Phase 3 will feel like common sense with notation attached.

This also connects to nearly everything already built in this series: Day 1's majority-class baseline, Day 3's correlation-based leakage checks, Day 4's z-score and IQR outlier rules and its class-stratified missingness check — all of it leans on the ideas in today's lesson, whether or not it was named explicitly at the time.

And it's worth knowing this going into interviews specifically: "explain a p-value to me" remains one of the most reliably mis-answered questions in the field, including by people who run statistical tests every week.

Mental Model

A statistic is a single spoonful you use to judge an entire pot of soup.

• The mean judges the soup by its average temperature across the whole pot — accurate, unless one corner is still boiling, in which case that one corner badly distorts your sense of everywhere else.

• The median judges the soup by tasting specifically from the middle of the pot — much harder for one weird corner to throw off.

• Sampling is the spoonful itself. You're never tasting the whole pot, only a spoonful, and the real question this lesson answers is exactly how much you should trust that one spoonful's verdict about the whole pot — and how that trust changes as your spoonful gets bigger.

History in 60 Seconds

• 1654 — Blaise Pascal and Pierre de Fermat's correspondence on gambling problems lays the formal groundwork for probability theory.

• 1713 — Jacob Bernoulli's Ars Conjectandi, published eight years after his death, proves the Law of Large Numbers — the formal idea behind "more data, more reliable estimate," which today's sampling simulation will show directly.

• 1809 — Carl Friedrich Gauss formalizes what we now call the normal distribution while modeling errors in astronomical measurements — the same bell curve quietly underlying a huge share of ML's default assumptions.

• 1900 — Karl Pearson introduces the chi-squared test, one of the first formal tools for asking "could this pattern have arisen by chance?"

• 1935 — Ronald Fisher publishes The Design of Experiments, popularizing the p-value and introducing Fisher's exact test — the same test today's hands-on example uses — born from a colleague's claim that she could tell, by taste, whether milk or tea was poured into a cup first.

Key Terminology Table

Core Concepts

Mean, Median, and Why the Difference Isn't Just Style

The mean is the sum of all values divided by how many there are. The median is whatever value sits in the middle once the data is sorted. They agree closely on symmetric, outlier-free data and diverge sharply the moment a distribution gets skewed or picks up an extreme value — which is exactly why Day 4 recommended median imputation for skewed columns like Titanic's fare. Today's hands-on example shows precisely how much one extreme value can move each statistic, with real numbers instead of a general warning.

Variance and Standard Deviation: Measuring How Spread Out Things Are

Variance averages the squared distance of each point from the mean. The squaring matters for two reasons: it stops positive and negative deviations from canceling out, and it penalizes large deviations more than small ones. Standard deviation is just the square root of variance, brought back to the original units so it's interpretable — "this column typically varies by about $50," not "this column has a variance of 2,500 dollars-squared." Z-scores, which Day 4 used for outlier detection, are nothing more than "how many standard deviations away from the mean is this point?"

Probability Distributions, and Why the Normal Distribution Keeps Showing Up

A probability distribution describes how likely different values or ranges of values are. The normal distribution — the symmetric bell curve — shows up constantly in statistics and ML not because the universe is obsessed with bells, but because of a specific, provable result: when you add up or average many small, mostly-independent effects, the result tends toward a normal distribution, regardless of the shape of the original effects. That result has a name, and it's the most load-bearing idea in this entire lesson.

Sampling and the Central Limit Theorem: Why More Data Means More Confidence

You almost never have access to an entire population — you have a sample. The Central Limit Theorem (CLT) says that if you take repeated samples and compute their means, those sample means form their own distribution — the sampling distribution of the mean — which approaches a normal shape and gets narrower as your sample size grows, no matter what shape the original data had.

That narrowing has an exact formula, not just a "more is better" intuition:

standard error = population standard deviation / sqrt(sample size)

Notice the square root. Quadrupling your sample size only halves your standard error, not quarters it. "More data helps" is true; "twice the data means twice the precision" is not, and today's hands-on example shows the real numbers behind that gap.

Hypothesis Testing and p-values: Telling Signal from Noise

A hypothesis test starts by assuming a null hypothesis — typically, "there's no real effect or difference here, what you're seeing is just sampling noise." The p-value is the probability of observing data this extreme, or more extreme, if that null hypothesis were actually true. A small p-value means your data would be surprising under the null hypothesis; a large one means it wouldn't be.

What a p-value is not: it is not the probability that the null hypothesis is true, and it is not a measure of how large or important an effect is. A tiny, meaningless effect can produce a tiny p-value if your sample is large enough, and a real, meaningful effect can produce a large p-value if your sample is small. Fisher himself popularized the conventional 0.05 threshold, and the field has been gently fighting its overuse as an automatic "true/false" switch ever since — a fight you'll see play out directly in this lesson's own results.

Visual Explanations

What the Central Limit Theorem actually claims, visually:

flowchart LR
    Pop[Population] -->|repeated random sampling| S1[Sample mean #1]
    Pop -->|repeated random sampling| S2[Sample mean #2]
    Pop -->|repeated random sampling| S3[Sample mean #3]
    S1 --> Dist[Sampling distribution of the mean]
    S2 --> Dist
    S3 --> Dist
    Dist -->|as sample size grows| Narrow[More normal-shaped, and narrower]

A quick reference for choosing a summary statistic:

And the decision flow behind every hypothesis test in this lesson:

flowchart TD
    H["State a null hypothesis:<br/>'there is no real difference'"] --> T[Run a statistical test,<br/>get a p-value]
    T --> Q{Is the p-value below<br/>your significance threshold?}
    Q -->|Yes| R["Reject the null —<br/>but this is evidence, not proof"]
    Q -->|No| F["Fail to reject —<br/>could mean no effect, or just not enough data to tell"]

Hands-On Example

Three real checks, building toward one specific payoff:

1. Mean vs. median sensitivity, using the Titanic fare column exactly as it exists — no synthetic outlier needed, since the dataset already has a genuinely extreme value in it.

2. A Central Limit Theorem simulation, repeatedly resampling from that same real fare data at different sample sizes, to see the standard error formula play out with real numbers instead of staying abstract.

3. Fisher's exact test, run on Day 4's exact open question: was the residual 1st-class survival gap between recorded and missing ages real, or was it noise from a small sample? We'll get an actual answer this time.

Environment Setup

python -m venv venv

source venv/bin/activate
# Windows
venv\Scripts\activate

pip install numpy pandas seaborn scipy

Complete Working Code

Every number in the breakdown below came from actually running this script.

"""
day_05_statistics_essentials.py
Mean/median sensitivity, a real CLT simulation, and a hypothesis test
that finally resolves Day 4's open question about Titanic's age column.
"""

import numpy as np
import pandas as pd
import seaborn as sns
from scipy.stats import fisher_exact

titanic = sns.load_dataset("titanic")

# ============================================================
# PART 1: MEAN vs MEDIAN SENSITIVITY (using real fare data)
# ============================================================
fare = titanic["fare"].dropna()

print("=== MEAN vs MEDIAN: FULL FARE DATA ===")
print(f"Mean fare:   {fare.mean():.2f}")
print(f"Median fare: {fare.median():.2f}")

fare_no_outlier = fare[fare < fare.max()]
print(f"\nAfter removing the single highest fare ({fare.max():.2f}):")
print(f"Mean fare:   {fare_no_outlier.mean():.2f}  (changed by {fare_no_outlier.mean() - fare.mean():+.2f})")
print(f"Median fare: {fare_no_outlier.median():.2f}  (changed by {fare_no_outlier.median() - fare.median():+.2f})")

# ============================================================
# PART 2: CENTRAL LIMIT THEOREM SIMULATION
# ============================================================
print("\n=== SAMPLING DISTRIBUTION OF THE MEAN (CLT simulation) ===")
print(f"True population mean fare: {fare.mean():.2f}")
print(f"True population std fare:  {fare.std():.2f}")

rng = np.random.RandomState(42)
for n in [5, 30, 100]:
    sample_means = [rng.choice(fare, size=n, replace=True).mean() for _ in range(5000)]
    theoretical_se = fare.std() / np.sqrt(n)
    print(f"n={n:4d}: simulated standard error = {np.std(sample_means):.2f}  "
          f"| theoretical (pop_std / sqrt(n)) = {theoretical_se:.2f}")

# ============================================================
# PART 3: RESOLVING DAY 4'S OPEN QUESTION WITH A REAL HYPOTHESIS TEST
# ============================================================
titanic["age_missing"] = titanic["age"].isnull().astype(int)

print("\n=== WAS DAY 4'S RESIDUAL CLASS GAP SIGNAL OR NOISE? ===")
for pclass in [1, 2, 3]:
    sub = titanic[titanic["pclass"] == pclass]
    table = pd.crosstab(sub["age_missing"], sub["survived"])
    odds_ratio, p_value = fisher_exact(table)
    print(f"Class {pclass}: p-value = {p_value:.4f}  "
          f"{'(below 0.05)' if p_value < 0.05 else '(not below 0.05)'}")

Running this prints:

=== MEAN vs MEDIAN: FULL FARE DATA ===
Mean fare:   32.20
Median fare: 14.45

After removing the single highest fare (512.33):
Mean fare:   30.58  (changed by -1.62)
Median fare: 14.45  (changed by +0.00)

=== SAMPLING DISTRIBUTION OF THE MEAN (CLT simulation) ===
True population mean fare: 32.20
True population std fare:  49.69
n=   5: simulated standard error = 22.40  | theoretical (pop_std / sqrt(n)) = 22.22
n=  30: simulated standard error = 9.07  | theoretical (pop_std / sqrt(n)) = 9.07
n= 100: simulated standard error = 4.94  | theoretical (pop_std / sqrt(n)) = 4.97

=== WAS DAY 4'S RESIDUAL CLASS GAP SIGNAL OR NOISE? ===
Class 1: p-value = 0.0654  (not below 0.05)
Class 2: p-value = 0.5432  (not below 0.05)
Class 3: p-value = 0.8146  (not below 0.05)

Code Breakdown

Mean vs. median. Removing exactly one passenger's fare — the single highest value in 891 rows — shifts the mean by $1.62. The median doesn't move by a single cent. That's not a rounding artifact; it's the literal definition of each statistic in action. One real data point. Two completely different reactions.

The CLT simulation. Look at how closely the simulated standard error matches the theoretical formula at every sample size: 22.40 simulated vs. 22.22 theoretical at n=5, an exact 9.07-to-9.07 match at n=30, and 4.94 vs. 4.97 at n=100. The formula isn't an approximation pulled from a textbook — it's exactly what 5,000 rounds of real resampling produces. Also notice the diminishing returns predicted in Core Concepts: going from n=5 to n=30 (6x the data) cuts standard error roughly in half, and going from n=30 to n=100 (about 3.3x the data) cuts it by less than half again — consistent with error shrinking as the square root of sample size, not in direct proportion to it.

Resolving Day 4. None of the three p-values clear the conventional 0.05 threshold — not even the 1st-class gap we'd flagged as not fully ruled out, which lands at 0.0654. That's genuinely close to the threshold, and it's tempting to read it as "almost significant." Resist that temptation: a p-value of 0.0654 means that if there were truly no association between age-missingness and survival in 1st class, we'd see a gap this large or larger about 6.5% of the time purely by chance. That's not strong evidence of a real effect, but with only 30 missing-age passengers in that class, it's also not strong evidence against one — the test simply doesn't have much statistical power at that sample size. The honest conclusion is the same one Day 4 already reached by inspection, now backed by an actual test: the original 11-point aggregate gap was mostly the passenger-class confound, and nothing in the class-stratified data clears the bar for calling the residual a real, distinct effect.

Common Mistakes

• Treating the mean as the automatically "correct" summary statistic. It's the right choice for symmetric, outlier-free data and a poor one otherwise — check both, as today's fare example shows directly.

• Assuming more data gives proportional precision gains. Standard error shrinks with the square root of sample size. Tripling your data does not triple your precision.

• Treating "not statistically significant" as "proven no effect." A p-value of 0.0654 is not evidence of zero effect — especially with a small sample, it can just as easily mean "not enough data to tell," which is exactly the situation in Day 4's 1st-class result.

• Misreading the p-value itself as "the probability the null hypothesis is true." It isn't. It's the probability of data this extreme assuming the null hypothesis is true — a completely different statement.

• p-hacking — testing many things and reporting only the one that came out significant. At a 0.05 threshold, roughly 1 in 20 truly null tests will look significant by chance alone; testing enough things guarantees you'll find one.

• Confusing statistical significance with practical significance. A large enough sample can make a genuinely trivial difference "statistically significant" — always ask how big the effect actually is, not just whether a p-value cleared a threshold.

Best Practices

• Report both the mean and median when summarizing a new numeric column — a large gap between them is itself a useful signal about skew before you've made a single chart.

• Decide your significance threshold and what counts as a meaningful effect size before running a test, not after seeing the result.

• Report an effect size or confidence interval alongside any p-value — "p=0.03" alone tells you almost nothing about how big or important the effect is.

• When a result isn't significant, explicitly check whether you simply lack the statistical power to detect a real effect, rather than concluding the effect doesn't exist.

Production Perspective

A/B testing platforms are, underneath the dashboard, hypothesis-testing infrastructure running continuously — every "this button color increased signups" claim is a p-value with a UI wrapped around it, and the same misreadings covered in Common Mistakes (treating "not significant" as "no effect," stopping a test early the moment it looks significant) show up constantly in real product experimentation.

Statistical monitoring for deployed models leans on the same machinery: data drift detection often runs formal distributional tests (like the Kolmogorov-Smirnov test) under the hood to decide whether incoming data still resembles the training distribution, or has shifted enough to need attention — a preview of Phase 5's drift detection topic.

Sample size and statistical power translate directly into business resourcing decisions: how many users, and how many days, does an experiment actually need before a result can be trusted? Running an underpowered experiment and shipping a decision off a falsely "significant" result has a real cost; so does running an overpowered one for weeks longer than necessary, burning a/b testing capacity that other experiments needed.

Real-World Applications

• A/B testing for product, pricing, and UX changes — the most common real-world home for hypothesis testing in tech.

• Clinical trials, which essentially invented modern hypothesis testing as a discipline.

• Manufacturing quality control, using statistical process control charts built on the same mean/variance ideas from today.

• Survey research and polling, where "margin of error" is just a standard error wearing a more public-facing name.

• Model and data monitoring in production, where distributional tests decide whether something has actually changed or whether you're looking at noise.

Interview Questions

1. Explain a p-value in plain language, and explain what it does not mean. A p-value is the probability of seeing data this extreme or more, if the null hypothesis were actually true. It is not the probability that the null hypothesis is true, and it says nothing on its own about how large or important an effect is.

2. When would you prefer the median over the mean as a summary statistic? When the data is skewed or contains outliers that would otherwise pull the mean away from a typical value — Titanic's fare column, and Day 4's broader argument for median imputation, are direct examples.

3. Explain the Central Limit Theorem and why it matters for the reliability of an average. It states that the sampling distribution of a mean approaches a normal shape and narrows as sample size grows, regardless of the underlying data's shape. It matters because it's the formal justification for why larger samples give more reliable, less variable estimates.

4. What's the difference between standard deviation and standard error? Standard deviation measures spread within a single dataset. Standard error measures how much a statistic (like a sample mean) would vary if you repeated the sampling process — it shrinks as sample size grows, while standard deviation generally doesn't.

5. Why might "statistically significant" not mean "practically important"? Statistical significance depends partly on sample size — a large enough sample can make a tiny, real-world-irrelevant effect register as significant. Practical importance depends on the size of the effect itself, which a p-value alone doesn't communicate.

6. If a test gives you p=0.06, what would you actually do next? Avoid treating it as a clean failure. Check the sample size and consider whether the test was underpowered, look at the effect size and confidence interval rather than the p-value alone, and decide whether collecting more data is feasible before drawing a final conclusion.

7. What's the difference between Type I and Type II error? A Type I error is rejecting a true null hypothesis — a false positive. A Type II error is failing to reject a false null hypothesis — a false negative, often caused by insufficient statistical power.

8. Why isn't a larger sample size a substitute for fixing biased sampling? The Central Limit Theorem reduces random sampling error, but it does nothing to correct systematic bias in how the sample was collected — a biased sample just becomes a more confidently wrong estimate as it grows, not a more accurate one.

Self-Assessment

• Can you explain, using today's fare numbers, why removing one data point changed the mean but not the median?

• Could you explain to a non-technical person why "not statistically significant" doesn't mean "definitely no effect"?

• Can you compute and correctly interpret a p-value for a dataset you've worked with?

• Can you explain why Day 4's leftover question needed an actual test to answer, rather than just a confident guess from eyeballing the numbers?

Portfolio Challenges

Beginner Challenge

Compute the mean, median, and standard deviation for the Titanic age column, then check how much the mean shifts if you remove the single oldest passenger. Compare the size of that shift to today's fare example.

Intermediate Challenge

Re-run the CLT simulation using age instead of fare, with the same three sample sizes. Report whether the simulated standard errors match the theoretical population_std / sqrt(n) formula as closely as they did for fare, and explain any difference you find.

Advanced Challenge

Pick a different stratification question in the Titanic dataset — for example, whether embarked's missingness relates to survival once you control for class — and run Fisher's exact test on it. Report and honestly interpret the p-value, explicitly discussing whether your sample size gives the test enough power to detect a real effect if one exists.

Advanced Insights

Every "Model A beat Model B by 2 percentage points" claim you'll encounter from Phase 3 onward is implicitly a hypothesis-testing question, and most practitioners skip the rigor entirely and just eyeball the difference. Knowing how to attach an actual test — and a sense of whether the comparison even had enough statistical power to be trustworthy — is a real, practical edge most people building models never develop.

It's also worth knowing about the multiple comparisons problem before you go looking for more examples like today's: if you test many things at once (many features, many hyperparameter settings, many model variants), some will look "significant" by chance alone, purely from running enough tests. This is a quieter, more common version of the same trap that produces flashy, irreproducible findings elsewhere — and it's a direct preview of why cross-validation and careful train/validation/test separation, both coming in Phase 3, exist in the first place.

Key Takeaways

• Mean and median can tell very different stories about the same data — the size of the gap between them is itself a useful, free diagnostic for skew and outliers.

• The Central Limit Theorem comes with an exact formula, standard error = population std / sqrt(n), which today's simulation matched almost exactly — and which explains why more data has diminishing, not proportional, returns.

• A p-value tells you how surprising your data would be under a no-effect assumption. It does not tell you the probability that there's no effect, and it does not tell you whether an effect is large enough to matter.

• Day 4's open question is now closed: none of the within-class survival gaps from missing age data clear the bar for statistical significance, not even the closest one. The original signal really was mostly the passenger-class confound, exactly as Day 4 suspected.

What's Next

Day 6 moves from statistics to the other branch of math every ML algorithm leans on: Linear Algebra Essentials — vectors, matrices, and dot products, taught the same way as today: just enough rigor to make the algorithms in Phase 2 make sense, no more.

References

Beginner

• Google: Machine Learning Crash Course — includes a module on the statistical foundations used throughout this lesson.

• SciPy: scipy.stats documentation — official reference for fisher_exact and the broader statistical testing tools used in this article's code.

Intermediate

• Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani, An Introduction to Statistical Learning — free at statlearning.com, with a solid grounding in the statistical inference ideas this lesson builds on.

Professional

• Aurélien Géron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (O'Reilly) — covers the practical statistical groundwork behind model evaluation in depth. Available through O'Reilly and major booksellers.

Advanced

• Ronald A. Fisher, The Design of Experiments (Oliver and Boyd, 1935) — the original source of Fisher's exact test, used directly in today's hands-on example, and of the p-value convention this lesson both uses and pushes back on.