Neural Networks, Explained Simply - Part 3: Activation Functions and the Problem with Straight Lines

You've probably felt this at a party: too few people and it's awkward, too many and it's too loud to talk to anyone. There's a sweet spot in the middle, and no straight line can capture that shape.

TL;DR: A plain weighted sum can only ever draw straight-line-shaped decisions. Real preferences bend and curve, so every neuron passes its total through a small "bending" rule called an activation function before passing it along. That's the piece we mentioned but didn't explain back in Part 1.

The straight-line problem

Here's a subtlety we skipped past in Part 1. If a neuron's output were just its raw weighted sum, passed straight through with no bending at all, stacking many of those would still collapse into one giant straight line. Adding straight lines together, however many times, only ever gives you another straight line. Depth wouldn't buy you anything.

Part 1's neuron avoided that specific trap with its hard on/off cutoff (go or don't go, nothing in between). But that cutoff creates a different problem, one that connects directly to Part 2. Training works by nudging each weight a little, based on how far off a guess was. A light switch doesn't have a "little": it's either on or off, with no middle ground to nudge toward. That makes it nearly impossible to work out which direction to adjust the weights feeding into it.

What we actually need is something in between: a rule that's still bent enough to keep depth meaningful, but keeps enough of a slope that "nudge it a little" actually means something.

Enter the activation function

An activation function is that in-between rule: a small math function that reshapes a neuron's raw number before it moves to the next layer, in a way that's both nonlinear (so stacking layers keeps adding power) and, unlike an on/off switch's instant jump, changes gradually enough to give training an actual direction to nudge toward.

Here are two common ones:

Sigmoid takes any number, however large or negative, and squashes it into a smooth range between 0 and 1. Think of it as a dimmer switch instead of an on/off light switch: it can express "70% confident," not just "yes" or "no."

ReLU (short for Rectified Linear Unit) is simpler: if the number's negative, treat it as 0. If it's positive, let it through exactly as is. It's basically a one-way valve. Despite being this simple, it's the default choice in most modern networks, because it's cheap to compute and, once you stack enough of these, the network can approximate remarkably complex curves.

Back to the party

With ReLU-style neurons, a network can build detectors like "start ramping up once there are more than 3 friends" and "start ramping down twice as fast once there are more than 15 friends." Add those together, and the rise turns into a fall right where the second detector kicks in, producing exactly the sweet-spot shape a single straight-line rule could never draw:

This is the real reason activation functions matter: they're not a technical footnote, they're what makes "network" mean something more powerful than "one big weighted sum."

Quick recap

• If a neuron's output were just its raw weighted sum passed straight through, stacking layers would still collapse into one straight line, no matter how deep the network.

• Part 1's hard on/off cutoff avoids that, but creates a different problem: training nudges weights a little at a time, and a light switch has no "little" to nudge toward.

• An activation function solves both: it's nonlinear enough to make depth meaningful, and, unlike an on/off switch's instant jump, changes gradually enough to give training a direction to nudge toward.

• Sigmoid squashes any number into a smooth 0-to-1 range; ReLU zeroes out negatives and passes positives through unchanged.

• Stacking activation-bent neurons lets a network learn genuine curves, like a rise-then-fall sweet spot, not just a single straight-line rule.

Try it yourself

Think of another decision with a sweet spot, not just a yes/no: coffee (too little and you're groggy, too much and you're jittery), or study time before a test (too little and you're unprepared, too much and you're exhausted). Sketch what that curve would look like on paper, then notice: a straight line could never draw it, but two or three "bend points" could.

Next up: Loss Functions and Gradient Descent, where we'll finally explain, in plain language, exactly how a network decides which direction to nudge each weight during training.