--- title: "nginx Load Balancing: A Hands-On Guide With a Working Lab" author: "Samod Alex" published: "2026-11-15T05:49:12.329Z" updated: "2026-11-15T05:49:12.329Z" url: "https://zyvop.com/nginx-load-balancing-a-hands-on-guide-with-a-working-lab-k5d1h" canonical: "https://zyvop.com/nginx-load-balancing-a-hands-on-guide-with-a-working-lab-k5d1h" tags: ["Docker","Load Balancing","DevOps","Reverse Proxy","nginx"] --- # nginx Load Balancing: A Hands-On Guide With a Working Lab Pick the right method, handle failures and keep connections healthy, with a Docker lab where you can watch nginx balance requests. ![Cover image]() A load balancer spreads requests across several copies of your app, and nginx can do that job in about ten lines of config. Surviving a failure takes a few more, because the defaults leave gaps. Left alone, nginx can wait up to a minute to connect to a dead server, and it only learns a server is down when a real user's request fails. That leaves three questions: - Which method should pick the server? - How does nginx decide a server is dead? - What happens to a request whose server dies halfway through? In this guide we build a small lab with Docker Compose, three backends behind nginx, and use it to try each idea. The directives are checked against the official nginx documentation, and I note the version where a feature needs a recent release. At the time of writing, nginx stable is 1.30 and mainline is 1.31. ## The 30-second version You list your servers in an `upstream` block and point `proxy_pass` at it. nginx picks a server for each request. If talking to that server fails, nginx passes the request to the next one. ```mermaid flowchart LR client["Clients"] --> nginx["nginx, upstream app"] nginx --> app1["app1"] nginx --> app2["app2"] nginx --> app3["app3"] ``` Out of the box, the picking is weighted round robin, which means servers take turns, and a server with a higher `weight` takes more turns. Everything else in this post is a way to change who gets the next request, or what happens when a server misbehaves. ## The lab You need Docker with Compose. The backend is a 30-line Python server that answers with its own name, can be made slow with an environment variable, and echoes request headers at `/headers`: ```python # app.py import os import time from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer NAME = os.environ.get("NAME", "app") DELAY = float(os.environ.get("DELAY", "0")) # seconds to hold every request PORT = int(os.environ.get("PORT", "8000")) class Handler(BaseHTTPRequestHandler): protocol_version = "HTTP/1.1" # lets nginx reuse connections def do_GET(self): time.sleep(DELAY) if self.path == "/headers": # show what nginx forwarded to us body = "".join(f"{k}: {v}\n" for k, v in self.headers.items()) else: body = f"{NAME}\n" data = body.encode() self.send_response(200) self.send_header("Content-Type", "text/plain") self.send_header("Content-Length", str(len(data))) self.end_headers() self.wfile.write(data) def log_message(self, *args): # keep container logs quiet pass ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever() ``` Compose runs three copies of it plus nginx. The `APP3_DELAY` variable lets us make `app3` slow later without editing the file: ```yaml # docker-compose.yml x-app: &app image: python:3.12-alpine command: python /app/app.py volumes: - ./app.py:/app/app.py:ro services: app1: <<: *app environment: NAME: app1 app2: <<: *app environment: NAME: app2 app3: <<: *app environment: NAME: app3 DELAY: "${APP3_DELAY:-0}" # set APP3_DELAY=1 to make app3 slow nginx: image: nginx:stable ports: - "8080:80" volumes: - ./default.conf:/etc/nginx/conf.d/default.conf:ro depends_on: - app1 - app2 - app3 ``` The nginx config starts as plain round robin. The `log_format` is the important part: `$upstream_addr` shows which backend actually served each request, and `$upstream_status` shows what it answered. Those two variables make every experiment in this post visible. The file goes in `conf.d`, which the official image includes inside the `http` block, so `upstream` and `log_format` are in the right place. ```nginx # default.conf log_format lb '$remote_addr -> $upstream_addr status=$upstream_status ' 'rt=$upstream_response_time "$request"'; upstream app { server app1:8000; server app2:8000; server app3:8000; } server { listen 80; access_log /var/log/nginx/access.log lb; location / { proxy_pass http://app; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } ``` Start it and count where 12 requests land: ```bash docker compose up -d for i in $(seq 1 12); do curl -s localhost:8080; done | sort | uniq -c ``` You should see the three names with four requests each: ``` 4 app1 4 app2 4 app3 ``` After any config change, check it and reload: ```bash docker compose exec nginx nginx -t docker compose exec nginx nginx -s reload ``` Run `nginx -t` every time. A typo that stops nginx from loading is much easier to catch there than in production. When you are done with the lab, `docker compose down` removes it. ## Weights A `weight` sets a server's share of the traffic. The default is 1. ```nginx upstream app { server app1:8000 weight=3; server app2:8000; server app3:8000; } ``` Now `app1` takes 3 of every 5 requests. Reload, run the counting loop again with `seq 1 15`, and you should see 9 requests for `app1` and 3 each for the others. Use weights when servers are not equal, such as a bigger machine next to smaller ones, or when you want to send a canary a small share of traffic. ## Choosing a method Round robin is a fine default, and the right answer more often than people expect. Reach for something else only when you have a reason. ```mermaid flowchart TD start["Must one client keep hitting the same server?"] start -->|"No"| uneven["Are request durations very uneven, or connections long-lived?"] uneven -->|"No"| rr["Round robin, the default"] uneven -->|"Yes"| lc["least_conn"] start -->|"Yes"| key["Do you have a good key, like a session cookie or a URL?"] key -->|"Yes"| hash["hash key consistent"] key -->|"No"| ip["sticky cookie, or ip_hash"] ``` ### least\_conn Sends each request to the server with the fewest active connections, taking weights into account. It helps when some requests are much slower than others, such as uploads, reports or WebSockets, because a plain round robin keeps feeding a server that is already busy. ```nginx upstream app { least_conn; server app1:8000; server app2:8000; server app3:8000; } ``` You can see the difference in the lab: 1. Make `app3` slow, then send 60 requests, 10 at a time, with the round robin config you have now. 2. Add `least_conn;` to the `upstream` block and reload. 3. Send the same 60 requests again. ```bash APP3_DELAY=1 docker compose up -d app3 # run this line in steps 1 and 3 seq 60 | xargs -P 10 -I{} curl -s localhost:8080 | sort | uniq -c ``` With round robin, each server gets an equal share even though `app3` is holding connections for a second each. With `least_conn`, `app3` should end up with noticeably fewer requests, because it is busy and the others are not. ### ip\_hash Maps a client IP address to a server, so the same client keeps landing on the same one. It uses the first three octets of an IPv4 address, or the whole IPv6 address, so every client in the same /24 network goes to the same server. If you need to take a server out of an `ip_hash` group temporarily, mark it `down` instead of deleting it, so the existing mapping is preserved. ### hash Does the same with any key you choose. The `consistent` parameter matters when you add or remove servers: without it, most keys can move to a different server, and with it only a few do. That is what you want in front of caches. ```nginx upstream app { hash $request_uri consistent; server app1:8000; server app2:8000; server app3:8000; } ``` If the key is empty, for example a cookie the client doesn't have yet, nginx falls back to round robin for that request. ### random two Picks two servers at random, then sends the request to the one with fewer connections. It gets most of the benefit of `least_conn`, and it avoids several nginx instances all picking the same "least busy" server at once. ```nginx upstream app { random two least_conn; server app1:8000; server app2:8000; server app3:8000; } ``` ### least\_time Prefers the server with the lowest average response time. It is available in open source nginx from mainline 1.31.0. On the 1.30 stable branch it needs NGINX Plus. ## Sticky sessions Sometimes a client must come back to the same server, for example because the app keeps sessions in memory. You have three options, and the best one is the last. ### Option 1: ip\_hash `ip_hash` is the simplest, with two weaknesses. Many users can share one IP address (an office, a mobile carrier), which skews the load. And if another proxy or CDN sits in front of nginx, the address nginx sees is the proxy's, so everyone looks like one client. ### Option 2: a sticky cookie A cookie solves both. Since nginx 1.29.6, and so in the 1.30 stable branch, the `sticky` directive is available in open source nginx: ```nginx upstream app { server app1:8000; server app2:8000; server app3:8000; sticky cookie srv_id expires=1h path=/ httponly samesite=lax; } ``` nginx sets the cookie on a client's first request and routes on it afterwards. If the chosen server can't handle a request, nginx picks another one as if the client had no cookie yet. The same release added a `drain` parameter for deployments: a server marked `drain` only receives requests from clients already bound to it, so you can take it out of rotation gradually. ### Option 3: no stickiness at all The best option is to avoid needing stickiness at all. Keep sessions in Redis or a database, or use signed tokens, and any server can handle any request. Then every server is interchangeable, and failover is trivial. ## When a server fails nginx's built-in failure detection is passive. It watches real traffic instead of probing servers on its own. Two parameters on each `server` control it: - `max_fails` is how many failed attempts, within `fail_timeout`, mark the server unavailable. The default is 1. - `fail_timeout` is both the window for counting failures and how long the server stays out of rotation. The default is 10 seconds. ```mermaid stateDiagram-v2 direction LR [*] --> Available Available --> Unavailable: max_fails failures within fail_timeout Unavailable --> Probing: fail_timeout elapsed Probing --> Available: a real request succeeds Probing --> Unavailable: the request fails ``` Note what the probe is: after `fail_timeout`, nginx tries the server again with a real client request. If it is still down, one user pays for that discovery, once per `fail_timeout`. This is the main limit of passive checks, and the reason active health checks exist (more on that below). Try it in the lab: ```bash docker compose stop app2 for i in $(seq 1 12); do curl -s localhost:8080; done | sort | uniq -c docker compose logs nginx | tail -20 ``` Every request still succeeds, now split between `app1` and `app3`. In the log, the request that discovered the failure shows two addresses in `$upstream_addr`, the dead one first. The first request may also be slow: nginx waited for the connection attempt to fail. ```mermaid sequenceDiagram participant C as Client participant N as nginx participant A2 as app2, down participant A3 as app3 C->>N: GET / N->>A2: connect A2--xN: connection error or timeout Note over N: counts a failure for app2 N->>A3: same request, next server A3-->>N: 200 N-->>C: 200 ``` If every server in the group fails, nginx has nowhere left to go and returns 502 Bad Gateway, or 504 Gateway Timeout when the attempts timed out. ### What counts as a failure This is decided by `proxy_next_upstream`, which goes in the `location` (or `server` or `http`) block, not in `upstream`. The default is `error timeout`: connection errors and timeouts count, and the request moves on to the next server. Other conditions you can add: - `http_502 http_503 http_504` (and `http_500`, `http_429`) treat those responses as failures. A 502 from a crashed backend usually means "try another server", so adding these is common. - `invalid_header` counts an empty or invalid response. - `http_403` and `http_404` can be listed to trigger a retry, but they are never counted toward `max_fails`. Three more settings keep retries safe and fast: ```nginx location / { proxy_pass http://app; proxy_connect_timeout 2s; proxy_next_upstream error timeout http_502 http_503 http_504; proxy_next_upstream_tries 3; proxy_next_upstream_timeout 10s; } ``` - `proxy_connect_timeout` defaults to 60 seconds. For a dead server that is a very long time to make a user wait. Two or three seconds is usually plenty on a healthy network. - `proxy_next_upstream_tries` and `proxy_next_upstream_timeout` cap how many servers one request may try and the total time spent trying. Both default to 0, which means no cap. Since nginx 1.9.13, requests with non-idempotent methods (POST, LOCK, PATCH) are not passed to the next server once they have been sent to a backend, because retrying could repeat a payment or a write. If your endpoints are safe to retry, you can add `non_idempotent` to `proxy_next_upstream`. Do that only when you are sure, ideally with idempotency keys. ### Spare servers and one server Mark a server `backup` and it only gets traffic when the primary servers are unavailable. Mark it `down` to take it out of rotation. The `backup` parameter cannot be combined with `hash`, `ip_hash` or `random`. ```nginx upstream app { server app1:8000 max_fails=3 fail_timeout=15s; server app2:8000 max_fails=3 fail_timeout=15s; server app3:8000 backup; } ``` Try it with this config: stop `app1` and `app2`, and `app3` takes over. One gotcha: if a group has a single server, `max_fails` and `fail_timeout` are ignored and that server is never considered unavailable. ## Share state between workers nginx runs several worker processes. Without a `zone`, each worker keeps its own copy of the run-time state, so failure counts and connection limits are per worker, not global. Add a shared memory zone to the group: ```nginx upstream app { zone app 64k; least_conn; server app1:8000; server app2:8000; server app3:8000; } ``` ## Connection reuse and protecting backends Opening a new TCP connection for every proxied request is wasteful. Since nginx 1.29.7, connections to upstream servers are kept alive by default, and `proxy_http_version` defaults to 1.1, so you get this for free on the current stable branch. On older versions you had to add it yourself: ```nginx upstream app { server app1:8000; keepalive 32; } location / { proxy_pass http://app; proxy_http_version 1.1; proxy_set_header Connection ""; } ``` The number after `keepalive` is the maximum count of idle connections cached per worker. It does not limit the total number of connections nginx opens to a backend. To cap that, use `max_conns` on the server, and put the group in a shared `zone` so the limit is global instead of per worker. ```nginx server app1:8000 max_conns=100; ``` ## Passing client information through Behind a proxy, your app sees nginx as the client. The `proxy_set_header` lines in the lab config pass the real details along. The Python backend can show you what it received: ```bash curl -s localhost:8080/headers ``` You should see `X-Real-IP`, `X-Forwarded-For` and `X-Forwarded-Proto` among the headers. Your app should read the client address from those, and only trust them when the request really came through your own proxy. ## Changing the pool without downtime Reloading with `nginx -s reload` is graceful: new workers start with the new config, and old workers finish their current requests before they exit. That covers adding, removing and re-weighting servers. One thing catches people out: nginx resolves the hostnames in an `upstream` block when it loads the config. If a name cannot be resolved at that moment, nginx refuses to start, which is why the lab uses `depends_on`. If a backend's IP changes later, nginx keeps using the old one until the next reload. Since nginx 1.27.3, open source nginx can re-resolve names while running. It needs a `zone`, a `resolver`, and the `resolve` parameter on the server: ```nginx upstream app { zone app 64k; resolver 127.0.0.11 valid=10s; server app1:8000 resolve; } ``` Here `127.0.0.11` is Docker's built-in DNS. Use your own DNS server's address elsewhere. A hostname that resolves to several IP addresses defines several servers at once. ## What still needs NGINX Plus The open source version covers a lot now. These are the features that still need the commercial NGINX Plus: | Feature | What it does | | --- | --- | | Active health checks (`health_check`) | Probes servers on a schedule, without waiting for a real request to fail | | `slow_start` | Ramps a recovered server's traffic up gradually | | Runtime API for upstreams | Adds, removes and re-weights servers without a reload | | Request queue (`queue`) | Holds requests briefly when every server is busy | Everything else in this post works in open source nginx, on the versions noted where a feature is recent. If you need active health checks on open source nginx, the usual workaround is to let your orchestrator remove unhealthy backends from the pool, then reload nginx. In Kubernetes, readiness checks do that job for you. ## A starting config for production Putting the pieces together, this is a sensible starting point for a stateless app. It uses the default round robin, so add `least_conn;` if your request times vary a lot: ```nginx upstream app { zone app 64k; server app1:8000 max_fails=3 fail_timeout=15s; server app2:8000 max_fails=3 fail_timeout=15s; server app3:8000 max_fails=3 fail_timeout=15s; } server { listen 80; location / { proxy_pass http://app; # plus the four proxy_set_header lines from the lab config proxy_connect_timeout 2s; proxy_next_upstream error timeout http_502 http_503 http_504; proxy_next_upstream_tries 3; proxy_next_upstream_timeout 10s; } } ``` Two reminders go with it. A single nginx is itself a single point of failure, so production setups run at least two behind a floating IP or a cloud load balancer. And keep nginx updated: the 1.30 stable branch has received several security fixes this year, so run the latest 1.30.x release rather than the first one you find in a package repository. ## Wrap-up Pick a method, and stay with round robin until something proves you wrong. Decide what counts as a failure, and keep timeouts short enough that users never wait on a dead server. Make the balancing visible with `$upstream_addr` in your logs, because a balancer you cannot observe is one you cannot trust. Not covered here: TLS termination, WebSocket upgrades, and balancing raw TCP or UDP with the `stream` module. Each is a post of its own. ## Further reading - [The]() `ngx_http_upstream_module` [reference]() - [The official HTTP load balancing guide]() - `proxy_next_upstream` [in the proxy module reference]() - [The nginx 1.30 changelog]()