CPU vs GPU vs TPU vs NPU: What Each Chip Is Actually For

A recent laptop or phone often has three chips that can run AI work: a CPU, a GPU, and a newer one called an NPU. The cloud servers behind chatbots use GPUs and, at Google, a fourth kind called a TPU. Every chip maker claims its processor is the one built for AI, so why do all four exist? Underneath the marketing, for AI work they are doing the same core job: multiplying large grids of numbers. What differs is how each one is built to do it, and that decides where each belongs.

This post explains what each processor is, what it is good and bad at, and how to choose between them, with one idea that cuts through the spec sheets: for much of modern AI, the limit is how fast you can move data, not how fast you can do math.

The short version

CPU: a few powerful, flexible cores. Best at varied, branching, latency-sensitive work. It runs your operating system, your databases, and the code that coordinates everything else.
GPU: thousands of simpler cores plus dedicated matrix hardware and very fast memory. The default workhorse for training and serving AI models.
TPU: Google's custom chip built around a grid of multiply-accumulate units. Very efficient at large matrix math, mostly rented through Google Cloud.
NPU: a small, power-frugal AI block inside a phone or laptop chip. Built to run modest models on-device without draining the battery.

Why AI workloads shape the hardware

A neural network is mostly matrix multiplication: take a block of input numbers, multiply by a block of learned weights, add the results, repeat billions of times. That work is regular and massively parallel, which is exactly what a general-purpose processor is not optimized for.

Two limits govern how fast it goes:

Compute: how many multiply-adds per second the chip can perform.
Memory bandwidth: how fast it can feed those units with weights and data.

Chip designers attack both, which is why the four processors look so different. They also live in different places:

flowchart TB
    subgraph DEV["Phone or laptop: one chip (SoC) sharing memory"]
        C1["CPU<br/>general tasks"]
        G1["GPU<br/>graphics, some AI"]
        N1["NPU<br/>always-on, low-power AI"]
        M1[("Shared memory")]
        C1 --- M1
        G1 --- M1
        N1 --- M1
    end
    subgraph DC["Data center: a host plus accelerators"]
        H["Host CPU<br/>orchestrates the job"] --> A1["GPUs<br/>training and inference"]
        H --> A2["TPUs<br/>Google Cloud"]
    end

CPU: the generalist

A CPU has a modest number of large, sophisticated cores, from a handful in a laptop to over a hundred in a server. Each core has deep caches, branch prediction, and out-of-order execution, all aimed at finishing one varied task as fast as possible.

Good at: operating systems, databases, web servers, business logic, anything with lots of if statements and unpredictable memory access. In AI, CPUs run classical machine learning (such as gradient-boosted trees), data preprocessing and tokenization (splitting text into the pieces a model reads), small or heavily quantized models (compressed to lower numeric precision), and the orchestration code that keeps accelerators busy. Modern server CPUs add vector and matrix extensions, such as AVX-512 and Intel's AMX, to speed up AI math.

Weak at: raw parallel throughput and memory bandwidth. A typical server CPU talks to ordinary DDR memory, which is far slower than the high-bandwidth memory on an accelerator. For big models, that gap dominates.

GPU: the default for AI

A GPU started as a graphics chip, and graphics is also lots of matrix and vector math. Instead of a few big cores, a GPU has thousands of small ones that execute the same instruction across many data elements at once. Data-center GPUs add tensor cores (NVIDIA's name for units dedicated to matrix multiplication), high-bandwidth memory (HBM) stacked next to the chip, and fast links such as NVLink to join many GPUs together.

The numbers show how much the design is about memory. NVIDIA's Blackwell B200 carries 192 GB of HBM3e at roughly 8 TB/s. Its successor, Rubin, is specified with 288 GB of HBM4 and about 22 TB/s of bandwidth, and NVIDIA said in July 2026 that it was in full production and being set up at its major customers. Specs like these change every year or two, so check current figures before relying on them.

Good at: both training and inference, plus graphics and scientific computing. GPUs are flexible: if you can express work as large parallel operations, a GPU can run it, and the software ecosystem (CUDA, NVIDIA's programming platform; ROCm, AMD's equivalent; and the major ML frameworks) is the broadest of the four.

Weak at: cost and power draw at the high end, supply constraints, and dependence on a vendor's software stack. Tiny or highly branchy workloads waste a big GPU.

TPU: Google's matrix machine

A Tensor Processing Unit is a custom chip (an ASIC) that Google designed specifically for neural-network math. Its defining feature is the systolic array: a grid of small multiply-and-add units, each passing its result to the next, so data streams through the chip instead of bouncing back to memory at every step. Each value fetched from memory is reused many times before anything is written back.

Google's seventh generation, Ironwood, shows the scale. Each chip has 192 GB of HBM and 7.37 TB/s of memory bandwidth, and Google offers configurations of up to 9,216 chips in a single system. Google positions it for large-scale training as well as inference. In April 2026 Google unveiled an eighth generation that splits the line in two: TPU 8t for training and TPU 8i for inference.

Good at: large, regular workloads such as big transformer models, where the array stays full. Google and its large customers point to strong efficiency and cost for that kind of sustained job, and major AI labs, including Anthropic and Meta, have signed large multi-year TPU deals. The pod-scale interconnect is built for huge clusters. You program TPUs mainly through JAX or PyTorch, with Google's XLA compiler, which translates model code into instructions the TPU can run, and its software stack underneath.

Weak at: irregular or branchy workloads, small batches, and portability. You mostly get TPUs by renting them from Google Cloud, though Google said in April 2026 that it will also sell them to a select group of customers for their own data centers. The tooling ecosystem is smaller than the GPU world's. Other companies have built similar custom accelerators, such as AWS Trainium and Meta's MTIA, so "TPU" is Google's brand for a wider idea.

NPU: AI at the edge

A Neural Processing Unit is a compact accelerator inside a phone or laptop chip, designed to run neural networks at very low power. Examples include Apple's Neural Engine, Qualcomm's Hexagon NPU, and the NPUs in Intel Core Ultra and AMD Ryzen AI processors. It sits beside the CPU and GPU and usually shares the same system memory.

Its job is the steady, small AI tasks you notice indirectly: background blur and noise removal on calls, live captions, photo and video enhancement, on-device search and assistants. Running them locally keeps your data on the device, and doing it on the NPU rather than the CPU or GPU saves battery.

NPU marketing leans on TOPS (trillions of operations per second). Microsoft's Copilot+ PC program requires an NPU with at least 40 TOPS. Qualcomm says its Snapdragon X2 Elite NPU reaches 80 TOPS, while AMD's Ryzen AI 300 series is quoted at up to 50 TOPS. Treat these as peak numbers measured by the vendor, usually at low-precision integer math. They don't predict how fast a real app will run, because memory bandwidth and software support matter just as much.

Good at: efficient, always-on, small-to-medium model inference.

Weak at: large models, training, and software consistency. Each vendor has its own toolchain, so developers often rely on portable layers such as ONNX Runtime, a cross-vendor engine for running models, or each platform's native framework.

Side by side

	CPU	GPU	TPU	NPU
Design goal	Fast, flexible single-task execution	High parallel throughput	Efficient large-scale matrix math	Low-power on-device AI
Cores	Few, powerful	Thousands, simple, plus tensor cores	Large matrix units (systolic arrays)	Small dedicated matrix engines
Typical memory	System memory (DDR; low-power LPDDR in phones and laptops)	HBM (data center) or GDDR (graphics-card memory)	HBM	Shared system memory
Where you find it	Everywhere	Data centers, workstations, gaming PCs	Google Cloud, plus some customer data centers	Phones and laptops
Programmed with	Any language	CUDA, ROCm, ML frameworks	JAX or PyTorch (XLA underneath)	Vendor SDKs, ONNX Runtime
Best for	Control logic, classic ML, small models	Training and serving at any scale	Sustained large training and serving	Battery-friendly local inference
Main limit	Parallelism and bandwidth	Cost, power, supply	Flexibility and availability	Model size and software fragmentation

Memory bandwidth is the hidden spec

Generating text from a large language model is largely a memory problem. For a dense model, producing each token (roughly a word or part of a word) means reading essentially all of the model's weights from memory. That gives a rough ceiling for a single stream of text:

tokens per second ≤ memory bandwidth ÷ bytes of weights read per token

Two back-of-envelope examples for dense models, ignoring the KV cache (the model's stored working memory of the text so far), software overhead, and batching (handling several requests at once). In the table, 7B means 7 billion parameters, and 4-bit or 8-bit is the storage per parameter: half a byte or one byte.

Setup	Weights	Memory bandwidth	Ceiling
7B model, 4-bit, on a laptop (assumed)	about 3.5 GB	about 100 GB/s	about 29 tokens/s
70B model, 8-bit, on a B200-class GPU	about 70 GB	about 8,000 GB/s	about 114 tokens/s

These are my own illustrative estimates, not benchmarks, and the laptop bandwidth is an assumption. Real throughput is lower, and serving many users at once (batching) raises total output because one read of the weights serves many requests. Mixture-of-experts models activate only part of their weights for each token, so their ceiling is higher than total size alone would suggest. But the pattern explains why HBM is on most data-center accelerators and why a bigger TOPS number on a laptop NPU doesn't mean a faster chatbot.

How they work together

Real products use several of these chips at once. Here is a typical flow when you ask a phone assistant a question out loud. Details vary by product, and simple requests may never leave the device.

sequenceDiagram
    participant NPU as Phone NPU
    participant CPU as Phone CPU
    participant Host as Cloud host CPU
    participant Acc as GPUs or TPUs
    NPU->>CPU: Wake word heard, speech turned into text
    CPU->>Host: Send the request over the network
    Host->>Acc: Run the language model
    Acc-->>Host: Generated answer
    Host-->>CPU: Reply text
    Note over CPU: Shows or speaks the reply

The NPU does the cheap, constant work of listening and transcribing at very low power. The phone's CPU ties the app together and handles the network. The cloud host's CPU coordinates the job, and the GPUs or TPUs do the one step that needs enormous memory bandwidth: running the large language model. Each chip handles the part it was built for.

Which one should you use?

flowchart TD
    A["Where does the workload run?"]
    A -->|"Phone or laptop"| B{"Small model, always on, battery matters?"}
    B -->|Yes| C["NPU"]
    B -->|"No: heavier model or one-off job"| D["GPU, or CPU for small jobs"]
    A -->|"Data center or cloud"| E{"Large training or high-volume serving?"}
    E -->|Yes| F["GPU, or TPU if your stack supports it and you can get capacity"]
    E -->|"No: classic ML, preprocessing, light inference"| G["CPU"]

Common misconceptions

"NPUs will replace GPUs." They solve different problems. An NPU is built for a few watts on a device. A data-center GPU can draw on the order of a kilowatt and is built for giant models.
"A TPU is just a faster GPU." It is a different design with different trade-offs: more efficient on large regular workloads, less flexible elsewhere.
"More TOPS means faster." TOPS ignores memory bandwidth, precision, and software support.
"CPUs are irrelevant for AI." They run most of the surrounding system, and they handle plenty of small-model inference and classical ML.
"GPUs are only for graphics." Graphics is where they started, but today's data-center GPUs are built around AI workloads.

Wrapping up

Think of the four as points on a trade-off curve between flexibility and efficiency. The CPU can do anything but runs relatively little of it in parallel. The GPU does highly parallel work flexibly. The TPU gives up some flexibility for efficiency at scale. The NPU gives up scale for efficiency in your pocket. In practice they work together: a CPU coordinates, a GPU or TPU does the heavy lifting in the cloud, and an NPU handles the small, constant tasks on your device.