API Gateway vs Load Balancer vs Service Mesh: What’s the Difference and When Do You Need Each?

If you have ever sketched a modern architecture, you have probably drawn boxes labeled “load balancer,” “API gateway,” and “service mesh,” and then wondered whether you really need all three. They all sit in the path of network traffic, they all route requests, and they increasingly share the same underlying proxies. No wonder they get confused.

The good news is that each one solves a different problem. In this post, we’ll break down what a load balancer, an API gateway, and a service mesh each do, compare them side by side, look at where their features overlap, and finish with a practical way to decide what your system needs today.

The Short Answer

• Load balancer: spreads incoming traffic across multiple healthy instances of a service. Which copy of this service should handle the request?

• API gateway: the front door for API clients. It authenticates callers, enforces limits, and routes each request to the right service. Who is calling, are they allowed to, and where should this go?

• Service mesh: a dedicated layer that manages communication between your internal services. How do these services talk to each other securely and reliably?

North-South vs East-West Traffic

The easiest way to tell them apart is by the direction of the traffic they handle.

• North-south traffic enters or leaves your system: a browser, a mobile app, or a partner calling your APIs.

• East-west traffic stays inside: one microservice calling another.

Load balancers and API gateways mostly handle north-south traffic. A service mesh exists for east-west traffic.

flowchart TD
    C[Clients] -->|"north-south"| LB{{"Load balancer"}}
    LB --> GW[API gateway]
    GW --> MESH
    subgraph MESH["Service mesh: east-west traffic"]
        O[Orders] -->|"mTLS"| I[Inventory]
        O -->|"mTLS"| P[Payments]
    end

What Is a Load Balancer?

A load balancer distributes incoming connections or requests across a pool of servers so that no single instance is overwhelmed and a failed instance doesn’t take the service down. It continuously runs health checks and stops sending traffic to anything unhealthy.

Load balancers work at two levels:

• Layer 4 (transport): balances TCP or UDP connections using IP addresses and ports. Fast and content-agnostic.

• Layer 7 (application): understands HTTP, so it can route by host, path, or header.

The algorithms it uses to pick a server (round robin, least connections, and so on) are a topic of their own, and we cover them, along with health checks and Layer 4 vs Layer 7, in my load balancing guide.

Typical features: health checks, TLS termination, connection draining, and basic host- or path-based routing.

Examples: AWS Application and Network Load Balancers, Google Cloud Load Balancing, HAProxy, NGINX, and Envoy.

What it doesn’t do: a load balancer doesn’t know who the caller is in any business sense. It won’t enforce per-customer quotas, issue API keys, or transform request payloads.

You need one when: you run more than one instance of anything. This is the one component almost every production system has.

What Is an API Gateway?

An API gateway is a single entry point for API consumers. Instead of every client knowing about every backend service, clients talk to the gateway, and the gateway applies policy and forwards each request to the right place.

flowchart TD
    R[Incoming request] --> A[Authenticate]
    A --> L[Rate limit]
    L --> V[Validate and transform]
    V --> RT[Route]
    RT --> S[Backend service]

Typical features:

• Authentication and authorization (API keys, OAuth 2.0, JWT validation)

• Rate limiting, quotas, and throttling per client or plan

• Routing and API versioning

• Request and response transformation, and protocol translation (for example, REST to gRPC)

• Response caching

• Analytics, logging, and developer portals

Examples: Kong, Apigee, Amazon API Gateway, Azure API Management, Tyk, Traefik, and Envoy-based gateways.

What to watch out for:

• Bottleneck and single point of failure. Everything flows through it, so it must be scaled and made highly available, usually behind a load balancer.

• Business logic creep. A gateway is great for cross-cutting concerns. Once it starts containing real business rules, it becomes a hard-to-test monolith in disguise.

• Gateway sprawl. One gateway per team with no shared standards recreates the problem it was meant to solve.

You need one when: you expose APIs to external clients or partners and want consistent authentication, rate limiting, versioning, and analytics in one place.

What Is a Service Mesh?

A service mesh is infrastructure for service-to-service communication. Instead of each team re-implementing retries, timeouts, encryption, and metrics in every service (and every language), the mesh handles them at the network layer.

It has two parts:

• Data plane: proxies that sit next to your workloads and handle the actual traffic.

• Control plane: the brain that distributes configuration, policy, and certificates to those proxies.

Here is the classic sidecar layout:

flowchart LR
    CP["Control plane<br/>(policy, config, certificates)"]
    subgraph PodA["Orders pod"]
        AppA[Orders app] <--> ProxyA[Proxy]
    end
    subgraph PodB["Payments pod"]
        ProxyB[Proxy] <--> AppB[Payments app]
    end
    ProxyA <-->|"mTLS"| ProxyB
    CP -.-> ProxyA
    CP -.-> ProxyB

Typical features:

• Mutual TLS (mTLS): encrypted, authenticated traffic between services, with workload identities and automatic certificate rotation

• Traffic management: retries, timeouts, circuit breaking, and weighted traffic splitting for canary releases

• Observability: consistent metrics, logs, and traces for every call, with no code changes

• Policy: rules about which services may call which

Examples: Istio, Linkerd, Consul, Cilium, and Kuma. Istio’s sidecars, gateways, and waypoint proxies are Envoy-based (its ambient ztunnel is written in Rust), while Linkerd uses its own lightweight proxy.

Sidecars and ambient mode: the classic model runs a proxy container beside every application pod. Istio’s ambient mode, which reached general availability in Istio 1.24, offers a sidecarless alternative: a shared, lightweight node-level proxy (ztunnel) handles Layer 4 features such as mTLS, and optional Layer 7 proxies called waypoints are added only where you need HTTP-level features. Sidecar mode is still supported, so it is a genuine choice. Feature coverage differs between the two modes, so check Istio’s feature status page against your requirements.

What to watch out for:

• Operational complexity. A mesh is another distributed system to install, upgrade, secure, and debug.

• Resource and latency overhead. Extra proxies mean extra CPU, memory, and network hops.

• Overkill for small systems. With a handful of services, the cost often outweighs the benefit.

You need one when: you have many services, and you need uniform mTLS, consistent traffic policy, and deep visibility into service-to-service calls without touching application code.

One Request, Three Layers

Here is how a single request might travel through a system that uses all three:

sequenceDiagram
    participant C as Client
    participant LB as Load balancer
    participant GW as API gateway
    participant O as Orders
    participant P as Payments
    C->>LB: HTTPS request
    LB->>GW: Pick a healthy gateway
    Note over GW: Authenticate,<br/>rate limit, route
    GW->>O: Forward to Orders
    O->>P: Call Payments
    Note over O,P: Mesh proxies add mTLS,<br/>retries, and metrics
    P-->>O: Result
    O-->>GW: Response
    GW-->>LB: Response
    LB-->>C: Response

Side-by-Side Comparison

Where the Lines Blur

In practice the boundaries are fuzzy, and that is normal:

• A cloud load balancer may do path-based routing and even authenticate users with an identity provider.

• An API gateway load-balances across the instances of the services behind it.

• Many service meshes include their own ingress gateway for traffic entering the mesh.

• Many gateways and meshes run on the same proxy, often Envoy.

The Kubernetes Gateway API is pulling these worlds together. It is a standard set of resources (Gateway, HTTPRoute, and others) for configuring routing. HTTPRoute is generally available, and since version 1.1 the same API can also describe service mesh traffic through the GAMMA initiative, so one routing model can cover both ingress and mesh. Support varies by implementation, so check yours.

Here is a 90/10 canary split at the edge, attached to a Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: orders-canary
spec:
  parentRefs:
    - name: public-gateway
  hostnames:
    - api.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /orders
      backendRefs:
        - name: orders-v1
          port: 8080
          weight: 90
        - name: orders-v2
          port: 8080
          weight: 10

To apply the same idea to east-west traffic inside a supporting mesh, you attach the route to a Service instead of a Gateway:

parentRefs:
    - group: ""
      kind: Service
      name: orders

A note if you are on Kubernetes: the community ingress-nginx controller was retired in March 2026, meaning no further releases or security fixes. That applies to that specific controller, not to the Ingress API itself (which remains available but is feature-frozen) or to other NGINX-based controllers. The Kubernetes project points new work toward Gateway API.

Do You Need All Three?

Almost certainly not on day one. Most systems grow into these components in stages, adding each one when a specific pain appears.

flowchart TD
    A[Start with a load balancer] --> Q1{"Do external clients call your APIs?"}
    Q1 -->|Yes| GW[Add an API gateway]
    Q1 -->|No| Q2
    GW --> Q2{"Do many services need mTLS and shared traffic policy?"}
    Q2 -->|Yes| SM[Add a service mesh]
    Q2 -->|No| OK[Stop here and revisit as you grow]
    SM --> PILOT[Start with a few critical services]

Common setups look like this:

Common Pitfalls

• Adopting a mesh too early. If you have five services and a small team, a mesh may add more operational work than it removes. Libraries, a good gateway, and solid observability can carry you a long way.

• Putting business logic in the gateway. Keep it to cross-cutting concerns such as authentication, limits, and routing.

• Stacking retries. If the client, the gateway, and the mesh each retry, the multiplication can turn a small failure into a flood of traffic. Decide which layer owns retries.

• Forgetting the gateway is a critical path. Run multiple instances, monitor it closely, and test failure scenarios.

• Unclear ownership. Decide which team owns each layer and which policies live where, or you will end up with conflicting rules.

The retry problem is worth seeing in numbers. If every layer makes up to three attempts, one failing call can multiply quickly:

flowchart LR
    C["Client<br/>3 attempts"] --> G["Gateway<br/>3 attempts each"] --> M["Mesh proxy<br/>3 attempts each"] --> S["Failing service<br/>up to 27 calls"]

Final Thoughts

These three components are complements, not competitors. A load balancer keeps traffic spread across healthy instances. An API gateway controls how outside clients use your APIs. A service mesh makes the conversations between your internal services secure, observable, and resilient.

Start with the simplest thing that works: a load balancer. Add a gateway when you need to manage external API consumers. Add a mesh when the complexity of service-to-service communication is genuinely costing you time, security, or reliability. And because the tooling is converging around shared standards such as Gateway API, the choices you make today are easier to evolve tomorrow.

Which of the three is the hardest to operate in your stack, and why? Share your experience in the comments.