Neural Networks, Explained Simply - Part 2: How Neural Networks Actually Learn

Neural Network Series · Part 2 · ~5 min read

Last time, we built a neuron that could decide whether to go for a walk, but we hand-picked its weights and bias. Real networks don't get that shortcut. They start out clueless and get better the same way you got better at riding a bike: try, wobble, adjust, try again.

TL;DR: A network starts with random, bad guesses. Each time it's wrong, it checks how far off it was and nudges its numbers a little closer to correct. Repeat that thousands of times across thousands of examples, and the guesses get good. That whole process is called training.

Quick recap from Part 1

A neuron takes inputs, multiplies each by a weight, adds a bias, and checks the total against a threshold. Last time, we picked the weights and bias ourselves: × 3 for rain, × 1 for cold, and so on. That worked because we already knew the right answer for a walk decision.

But for something like recognizing a cat in a photo, nobody knows the right weights in advance: there's no formula for "cat-ness." So instead of guessing the weights ourselves, we let the network find them by practicing.

Start with a bad guess

Let's reuse our walk-deciding neuron, but this time it starts out clueless: every weight is a small random number, and the bias is 0.

We show it one real example: it's raining (1), it's not cold (1), and there's free time (1). The correct answer, based on what actually happened that day, was stayed home (0).

The network does its math with its random weights and lands on a positive total, so it guesses go for a walk (1). Wrong.

Turning "wrong" into a fix

Here's the key idea: the network doesn't just note that it was wrong; it works out how wrong, and which inputs are most to blame.

That nudge is small and cautious, not a full correction, just a step in the right direction. Do this once, and the network barely changes. Do it across thousands of examples, and the weights slowly settle into values that actually make good decisions.

So what's "backpropagation"?

You've probably heard this word thrown around. Here's the plain version: backpropagation is the method a network uses to work out exactly how much each individual weight contributed to a wrong answer, moving backward from the mistake through every layer. Think of it as assigning blame fairly, one layer at a time, so every weight gets nudged by the right amount, not too much, not too little.

The full math behind that blame-assignment deserves its own post. We'll open it up properly once we've covered a few more building blocks, starting with Part 3's activation functions. For now, the concept is what matters:

Wrong guess → figure out who's to blame → adjust → try again

Quick recap

• Networks don't start out knowing the right weights; they start random and learn through training.

• Training means: guess, compare to the correct answer, measure the error, and adjust the weights slightly.

• Backpropagation is the technique for figuring out how much to adjust each weight, working backward through the network.

• This cycle repeats across huge numbers of examples before a network gets genuinely good.

Try it yourself

Remember the coffee-ordering neuron from Part 1? Say it predicted "order coffee" (1), but you actually decided not to (0). Same idea as before: guess versus reality.

No math needed this time, just reason it through: which input ("feeling tired," "before noon," "already had one today") most likely pushed that guess too high, and should have its weight nudged down for next time?

Next up: Activation Functions: Why Networks Aren't Just Straight Lines, where we'll finally unpack that sigmoid/ReLU mention from Part 1.