How a GPU Actually Works

In 2008, at NVIDIA's NVISION show in San Jose, MythBusters hosts Adam Savage and Jamie Hyneman were asked to explain the difference between a CPU and a GPU. They did not bring a chip. They brought a robot.

The first machine fired paintballs one at a time and slowly drew a smiley face. Then a sheet came off a second machine: a mile of air hose feeding 1,100 barrels. When it fired, a pixelated Mona Lisa appeared on the opposite wall in a fraction of a second (bit-tech's report).

That stunt is this article in miniature. The first robot is a CPU: flexible, precise, one thing at a time. The second is a GPU: every barrel is simple, but all 1,100 fire together. Everything below is that idea plus the engineering needed to keep every barrel loaded.

The short version

• A CPU is built to finish one task as fast as possible. A GPU is built to finish an enormous number of tasks per second, even if each one takes longer.

• It gets there with thousands of simple arithmetic units, a scheduler that juggles hundreds of thousands of threads, and a memory system tuned for bandwidth rather than latency.

• The same design that shades a pixel also multiplies a matrix, which is why a chip built for video games became the engine of modern AI.

The same idea, in pixels

Run a game at 4K and 60 frames per second. Each frame has 3,840 × 2,160 ≈ 8.3 million pixels, so the chip must produce roughly half a billion pixels every second. Each pixel needs a color computed from lighting, shadows, textures and materials, which can take hundreds or thousands of arithmetic operations. Multiply it out and you are in the trillions of operations per second.

Here is the detail that makes this tractable: pixel number one does not need to know anything about pixel number two. The work is embarrassingly parallel. Every hardware decision in a GPU is a bet on that single property.

Two philosophies: latency vs. throughput

Imagine you must move 50,000 people across a city in an hour. A sports car is thrilling: it crosses town in minutes, but it carries two people at a time. A fleet of buses is slower per trip and far less glamorous, but each bus carries sixty people and there are hundreds of them. If the goal is people per hour, the buses win by a mile.

A CPU is the sports car. A modern CPU core spends most of its transistors on tricks that make one thread run fast: big caches, branch predictors that guess which way an if-statement will go, and out-of-order engines that reshuffle instructions on the fly. The goal is minimal latency for one stream of work. A big server CPU has on the order of a hundred of these cores.

A GPU is the fleet of buses. It strips out most of that machinery and spends the transistors on arithmetic units and the plumbing to keep them fed. The goal is maximal throughput: total work completed per second. A single GPU thread is slow and rather dumb. But a recent datacenter GPU has tens of thousands of arithmetic lanes and keeps hundreds of thousands of threads in flight at once.

Anatomy of a modern GPU

I use NVIDIA's terminology because it is the most widely documented (AMD equivalents are in a table below). The numbers are for the H100, with newer chips covered near the end.

GPU
 ├── Graphics Processing Clusters (GPCs)
 │    └── Streaming Multiprocessors (SMs)     <- 132 on an H100
 │         ├── 4 sub-partitions, each with:
 │         │     • a warp scheduler
 │         │     • 32 FP32 lanes (“CUDA cores”)
 │         │     • a tensor core
 │         ├── register file (256 KB)
 │         └── shared memory / L1 cache (up to 256 KB)
 ├── L2 cache (50 MB), shared by every SM
 └── Memory controllers → HBM stacks (80 GB, ~3.35 TB/s)

The Streaming Multiprocessor (SM) is the unit to understand. A GPU is essentially a few dozen to a couple hundred SMs stamped out side by side, each a small, self-contained parallel computer. When you hear that the H100 has 16,896 “CUDA cores,” that is 132 SMs × 128 lanes each.

A decoder ring for the vocabulary:

SIMT: one instruction, thirty-two threads

You program a GPU by writing a function for one thread and then launching it across thousands or millions of them. That function is called a kernel. Here is the “hello world” of GPU programming, adding two arrays:

__global__ void add(const float* a, const float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;  // my global index
    if (i < n) c[i] = a[i] + b[i];                  // one element per thread
}

// Launch for a million elements, 256 threads per block:
add<<<(n + 255) / 256, 256>>>(a, b, c, n);

Every thread computes its own global index i from its block and thread coordinates, then handles exactly one element. The hardware hands blocks of threads out to SMs as they become free.

Underneath, the SM does not run 256 independent threads. It groups them into warps of 32 and executes each warp in lockstep: one instruction is fetched and decoded once, and all 32 lanes apply it to their own data. NVIDIA calls this model SIMT (single instruction, multiple threads). It is a clever economy. One instruction fetch and decode serves 32 lanes of arithmetic, which is why the control logic can be so small.

The catch is divergence. If threads in the same warp take different sides of an if, the hardware cannot run both sides at once. It runs the if side with the other threads masked off, then the else side with the first group masked off. In the worst case, a warp where every thread wants something different runs up to 32× slower. Good GPU code keeps neighboring threads on the same path.

The central trick: hiding latency with parallelism

Fetching a number from GPU main memory takes several hundred clock cycles, an eternity for a chip that can do an operation every cycle. A CPU fights that delay with giant caches and clever guessing. A GPU does something simpler and more radical: it stops waiting.

Picture a kitchen with sixty orders in progress. The cook never stands staring at the oven. Whenever a dish goes in to bake, they turn to another order. The oven is slow, but the cook is never idle.

An SM works the same way, with warps as the orders. It can keep up to 64 warps (2,048 threads) in flight. Every cycle, its schedulers pick a warp that is ready and issue its next instruction. If that warp asks for data from memory, it steps to the back of the line and another warp takes its turn. By the time the schedulers come back around, the data has usually arrived.

Two details make this work:

• Switching is free. Every resident warp keeps its own slice of the huge register file, so changing warps costs zero cycles.

• You need plenty of warps. If only a few are around, there is nothing to switch to and the chip sits idle. Engineers call this occupancy: how full an SM's warp slots are with ready work.

If you like numbers (optional). An H100 holds about 270,000 threads at once (132 SMs × 2,048), and it needs roughly that many. By Little's Law, data in flight equals bandwidth × latency: about 3.35 TB/s × 400 ns ≈ 1.3 MB of loads outstanding at every moment, or hundreds of thousands of small requests.

The memory hierarchy is the real game

A GPU's arithmetic is absurdly fast. Its memory, by comparison, is slow. So for most real programs, speed is decided less by how well you compute and more by how well you move data.

Let's stay in the kitchen. The chip has four places to keep ingredients, each bigger and farther away than the last:

The art is keeping the cook's hands full without too many trips to the warehouse. Two ideas do most of the work.

1. Coalescing: shop in bulk. When the 32 threads of a warp ask for 32 neighboring numbers, the hardware fetches them in one wide scoop and uses nearly every byte. If they ask for numbers scattered across memory, each request drags in a whole cache line to use 4 bytes of it, and the effective speed of memory collapses. Same data, a fraction of the speed. Layout is performance.

2. Tiling: reuse what you carry in. Take matrix multiplication, the workhorse of AI. The naive approach has each thread fetch a row and a column from the warehouse for every single output, and it uses each number once. The tiled approach has a whole thread block carry a small square of each matrix to the counter (shared memory) once, then reuse every number in it 32 times before going back for the next tile.

If you like numbers (optional). A FLOP is one floating-point operation, such as an add or a multiply. A naive matrix multiply does about 0.25 FLOPs per byte loaded. An H100 delivers roughly 67 trillion FP32 FLOPs per second against 3.35 TB/s of bandwidth, so it needs about 20 FLOPs per byte to stay busy: the naive version is about 80× short. 32 × 32 tiles raise the ratio 32×, to around 8, closing much of the gap. Engineers call FLOPs per byte arithmetic intensity, and comparing it with the chip's own ratio is the roofline model: you hit either the compute ceiling or the memory ceiling first.

Almost every fast GPU kernel, from matrix multiply to attention, is a variation on the same move: bring data from slow memory to fast memory once, and do as much math on it as possible before it leaves.

Where it all began: the graphics pipeline

GPUs did not start as math engines. They started as machines for turning triangles into pixels, and that heritage is still in the silicon. A frame passes through four stages: vertex processing moves each triangle's corners from 3D space to screen positions, rasterization (fixed-function hardware) works out which pixels each triangle covers, fragment shading runs a small program per pixel to compute its color, and output merging (the ROPs) depth-tests, blends and writes the result. Fragment shading is where most of the arithmetic lives, and it is exactly the same-program, millions-of-independent-inputs workload that SIMT was built for. Around the SMs sit texture units that fetch and filter images in hardware and, on modern chips, ray-tracing cores that accelerate testing rays against triangles.

The history explains the design. The 1999 GeForce 256 was marketed as the first “GPU” because it moved transform and lighting into hardware. By 2006, the GeForce 8800 and CUDA had turned shaders into general-purpose processors anyone could program. Researchers noticed that neural networks looked a lot like pixel shading, lots of independent math over big arrays, and in 2012 AlexNet won the ImageNet competition having trained on two consumer GTX 580 cards. The deep learning boom was on.

Tensor cores: a chip within the chip

Neural networks are, to a first approximation, matrix multiplications stacked on top of each other. Recognizing this, NVIDIA added tensor cores in the Volta generation (2017): hardware units that multiply and accumulate small matrix tiles (D = A × B + C) in a single operation, rather than one scalar multiply at a time.

The payoff is enormous. On an H100, the regular FP32 lanes top out around 67 TFLOPS, while the tensor cores deliver roughly 1,000 TFLOPS (a petaFLOP) of dense BF16/FP16 math, and about double that in FP8. Two things make this possible: circuits that do a whole small matrix multiply per instruction, and lower precision. Neural networks tolerate coarse numbers, so the industry has stepped down from FP32 to FP16, FP8 and now 4-bit formats on the newest chips. Fewer bits per number means more math per transistor and more numbers per byte of bandwidth.

But notice what this does to the roofline. At about 1,000 TFLOPS and 3.35 TB/s, you need roughly 300 FLOPs per byte to keep the tensor cores busy. Compute got faster much more quickly than memory did, so the memory problem got worse.

Why generating text is memory-bound

When an LLM generates one token, it must read essentially every weight in the model once. A 70-billion-parameter model stored in 8-bit precision occupies 70 GB. At 3.35 TB/s, just reading those weights takes about 21 milliseconds, which caps a single stream at roughly 48 tokens per second regardless of how many FLOPs the chip has. (This ignores the attention cache, which only makes it worse.)

The fix is batching: serve many users at once, so the weights are read once and applied to many tokens. One pass over 70 GB serving 64 requests puts far more of the tensor cores to work than serving one. It is the tiling lesson again, at the scale of an entire data center: amortize every trip to slow memory over as much math as you can.

Beyond one chip

The memory in a datacenter GPU is HBM: stacks of DRAM on the same package as the GPU die, connected through a silicon interposer with thousands of tiny wires (a 5,120-bit interface on the H100). That width, not clock speed, is where the bandwidth comes from. When one GPU is not enough, links like NVLink (900 GB/s per H100) tie many together, and training splits the work by data, by matrix slice or by layer. Communication then becomes the new bottleneck, the same lesson at a larger scale. Energy is the other limit: an H100 draws up to 700 W, and fetching a byte from off-chip memory costs far more energy than doing arithmetic on it.

What newer chips change, and what they don't

Everything above uses the H100 because its numbers are well documented, but it is no longer NVIDIA's newest chip. As of October 2026, Blackwell is in volume production and its successor, Rubin, is arriving.

What stays the same. The structure (SMs full of warps, a memory hierarchy) and the three lessons of latency hiding, coalescing and tiling. A kernel that was memory-bound on an H100 is almost certainly still memory-bound on a B200.

What changes:

• Smaller numbers. Blackwell adds native 4-bit math, so the 70 GB model from earlier shrinks to roughly 35 GB in FP4.

• Bandwidth grows, but compute grows faster. The B200's bandwidth is about 2.4× the H100's, while its peak low-precision throughput rose roughly 4× (comparing FP4 with the H100's FP8). The memory wall got taller, which is why memory is a headline feature of each generation and Rubin's big step is HBM4. At 8 TB/s, the single-stream token ceiling for our 70 GB example rises from roughly 48 to roughly 115 per second, and FP4 roughly doubles it again.

The same trade-offs show up beyond NVIDIA. AMD's Instinct accelerators are GPUs built on the same ideas, and Google's TPUs use large matrix units instead of GPU cores but make the same bet: simple arithmetic units, enormous parallelism and high-bandwidth memory. The vocabulary changes. The physics does not.

A mental model you can carry around

When a GPU workload is slow, ask three questions:

1. Is there enough parallel work? Tiny problems cannot fill a big chip, and every kernel launch has microseconds of overhead.

2. Is memory feeding the math? Check access patterns and data reuse. Most kernels are memory-bound, so ask whether the arithmetic intensity is above what the chip needs to stay busy.

3. Are neighboring threads doing the same thing? Divergent branches and uncoalesced accesses waste lanes and bandwidth.

And a rule of thumb: GPUs are bad at serial, branchy, pointer-chasing code, at tiny workloads, and at anything bottlenecked on shuttling data over PCIe from the CPU. They are spectacular at big, regular, data-parallel math.

The big picture

A GPU is an extreme answer to a narrow question: what if we gave up on making any single thing fast, and arranged everything, from silicon to scheduling to memory, around keeping a vast number of simple units busy? Parallelism hides latency. The memory hierarchy protects bandwidth. Once you see the pattern, GPU architecture stops looking like a zoo of acronyms and starts looking like one idea applied relentlessly.

Back to the paintball gun. The Mona Lisa needed 1,100 barrels, but it also needed a mile of air hose to keep every one of them supplied. That is the real job of a GPU designer. The barrels, the arithmetic units, are the cheap part. The plumbing is the art.

Further reading

• Programming Massively Parallel Processors by David Kirk and Wen-mei Hwu

• Simon Boehm, “How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance”

• Horace He, “Making Deep Learning Go Brrrr From First Principles”

• Fabian Giesen, “A Trip Through the Graphics Pipeline 2011”

Sources

• H100 specifications: Glenn K. Lockwood's H100 notes

• B200 specifications: Jarvislabs, NVIDIA B200 specs

• B300 and Rubin figures: BIZON, NVIDIA GTC 2026 key announcements (a hardware reseller's summary of NVIDIA's announcements)

• MythBusters demo: bit-tech, Mythbusters finish Nvision in style

Figures are approximate and refer to the NVIDIA H100 SXM unless noted. Blackwell and Rubin figures are as reported by vendors and the trade press as of October 2026, and they vary by configuration, precision and sparsity. The principles carry over to every generation.