--- title: "Why Your App Is Slow (And It's Not the Database)" author: "ZyVOP" published: "2026-06-07T06:41:15.805Z" updated: "2026-08-16T03:45:00.123Z" url: "https://zyvop.com/why-your-app-is-slow-and-it-s-not-the-database-iz3zc" canonical: "https://zyvop.com/why-your-app-is-slow-and-it-s-not-the-database-iz3zc" tags: ["backend","postgresql","System Design","debugging","Architecture","performance","nodejs"] --- # Why Your App Is Slow (And It's Not the Database) The slow query log shows nothing. The database looks fine. Here's a systematic breakdown of what's actually costing you latency — with instrumentation, code, and diagrams. ![Cover image]() The reflex when an API is slow is to check the database. You add an index. You look at `EXPLAIN ANALYZE`. You scale up the instance. Nothing changes. This happens because the slow query log only tells you about query _execution time_ — it says nothing about the other four places where your request can bleed latency before a query even runs, or after it finishes. This post systematically covers all five root causes, how to instrument each one, and what to do when you find it. The order matters: start from the top. * * * ## The Request Lifecycle Before diagnosing, it helps to see the full path a database-backed request takes. Latency can accumulate at every transition. ```svg PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA4MDAgMzEwIiB3aWR0aD0iMTAwJSIgc3R5bGU9Im1heC13aWR0aDo4MDBweDtkaXNwbGF5OmJsb2NrO21hcmdpbjoxLjVyZW0gMDtmb250LWZhbWlseTpzeXN0ZW0tdWksc2Fucy1zZXJpZiI+CgogIAogIDxyZWN0IHdpZHRoPSI4MDAiIGhlaWdodD0iMzEwIiBmaWxsPSIjMGYxMTE3IiByeD0iMTIiLz4KCiAgCiAgPHRleHQgeD0iNDAwIiB5PSIzMiIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMyIgZmlsbD0iIzZiNzI4MCIgZm9udC13ZWlnaHQ9IjUwMCI+V0hFUkUgTEFURU5DWSBISURFUyBJTiBBIERBVEFCQVNFLUJBQ0tFRCBSRVFVRVNUPC90ZXh0PgoKICAKCiAgCiAgPHJlY3QgeD0iMzAiIHk9IjU2IiB3aWR0aD0iMTMwIiBoZWlnaHQ9IjY0IiByeD0iOCIgZmlsbD0iIzNiMGYwZiIgc3Ryb2tlPSIjZWY0NDQ0IiBzdHJva2Utd2lkdGg9IjEiLz4KICA8dGV4dCB4PSI5NSIgeT0iODQiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTMiIGZvbnQtd2VpZ2h0PSI2MDAiIGZpbGw9IiNmODcxNzEiPlBvb2wgd2FpdDwvdGV4dD4KICA8dGV4dCB4PSI5NSIgeT0iMTAyIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjExIiBmaWxsPSIjZmNhNWE1Ij5XYWl0aW5nIGZvciBjb25uZWN0aW9uPC90ZXh0PgoKICAKICA8bGluZSB4MT0iMTYwIiB5MT0iODgiIHgyPSIxODIiIHkyPSI4OCIgc3Ryb2tlPSIjMzc0MTUxIiBzdHJva2Utd2lkdGg9IjEuNSIgbWFya2VyLWVuZD0idXJsKCNhcnIpIi8+CgogIAogIDxyZWN0IHg9IjE4MyIgeT0iNTYiIHdpZHRoPSIxMzAiIGhlaWdodD0iNjQiIHJ4PSI4IiBmaWxsPSIjM2IwZjBmIiBzdHJva2U9IiNlZjQ0NDQiIHN0cm9rZS13aWR0aD0iMSIvPgogIDx0ZXh0IHg9IjI0OCIgeT0iODQiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTMiIGZvbnQtd2VpZ2h0PSI2MDAiIGZpbGw9IiNmODcxNzEiPkxvY2sgd2FpdDwvdGV4dD4KICA8dGV4dCB4PSIyNDgiIHk9IjEwMiIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMSIgZmlsbD0iI2ZjYTVhNSI+V2FpdGluZyBvbiBhbm90aGVyIHR4PC90ZXh0PgoKICAKICA8bGluZSB4MT0iMzEzIiB5MT0iODgiIHgyPSIzMzUiIHkyPSI4OCIgc3Ryb2tlPSIjMzc0MTUxIiBzdHJva2Utd2lkdGg9IjEuNSIgbWFya2VyLWVuZD0idXJsKCNhcnIpIi8+CgogIAogIDxyZWN0IHg9IjMzNiIgeT0iNTYiIHdpZHRoPSIxMzAiIGhlaWdodD0iNjQiIHJ4PSI4IiBmaWxsPSIjMDUyZTE2IiBzdHJva2U9IiMyMmM1NWUiIHN0cm9rZS13aWR0aD0iMS41Ii8+CiAgPHRleHQgeD0iNDAxIiB5PSI4MCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMyIgZm9udC13ZWlnaHQ9IjYwMCIgZmlsbD0iIzRhZGU4MCI+UXVlcnkgZXhlYzwvdGV4dD4KICA8dGV4dCB4PSI0MDEiIHk9Ijk4IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjExIiBmaWxsPSIjODZlZmFjIj5XaGF0IGxvZ3MgY2FwdHVyZTwvdGV4dD4KICA8dGV4dCB4PSI0MDEiIHk9IjExMyIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMCIgZmlsbD0iIzRhZGU4MCI+IG9ubHkgdGhpczwvdGV4dD4KCiAgCiAgPGxpbmUgeDE9IjQ2NiIgeTE9Ijg4IiB4Mj0iNDg4IiB5Mj0iODgiIHN0cm9rZT0iIzM3NDE1MSIgc3Ryb2tlLXdpZHRoPSIxLjUiIG1hcmtlci1lbmQ9InVybCgjYXJyKSIvPgoKICAKICA8cmVjdCB4PSI0ODkiIHk9IjU2IiB3aWR0aD0iMTMwIiBoZWlnaHQ9IjY0IiByeD0iOCIgZmlsbD0iIzNiMGYwZiIgc3Ryb2tlPSIjZWY0NDQ0IiBzdHJva2Utd2lkdGg9IjEiLz4KICA8dGV4dCB4PSI1NTQiIHk9Ijg0IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEzIiBmb250LXdlaWdodD0iNjAwIiBmaWxsPSIjZjg3MTcxIj5TZXJpYWwgYXdhaXRzPC90ZXh0PgogIDx0ZXh0IHg9IjU1NCIgeT0iMTAyIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjExIiBmaWxsPSIjZmNhNWE1Ij5Vbm5lY2Vzc2FyeSBzZXF1ZW5jaW5nPC90ZXh0PgoKICAKICA8bGluZSB4MT0iNjE5IiB5MT0iODgiIHgyPSI2NDEiIHkyPSI4OCIgc3Ryb2tlPSIjMzc0MTUxIiBzdHJva2Utd2lkdGg9IjEuNSIgbWFya2VyLWVuZD0idXJsKCNhcnIpIi8+CgogIAogIDxyZWN0IHg9IjY0MiIgeT0iNTYiIHdpZHRoPSIxMjgiIGhlaWdodD0iNjQiIHJ4PSI4IiBmaWxsPSIjM2IwZjBmIiBzdHJva2U9IiNlZjQ0NDQiIHN0cm9rZS13aWR0aD0iMSIvPgogIDx0ZXh0IHg9IjcwNiIgeT0iODQiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTMiIGZvbnQtd2VpZ2h0PSI2MDAiIGZpbGw9IiNmODcxNzEiPkV4dGVybmFsIGNhbGw8L3RleHQ+CiAgPHRleHQgeD0iNzA2IiB5PSIxMDIiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiNmY2E1YTUiPk5vIHRpbWVvdXQgc2V0PC90ZXh0PgoKICAKICA8ZGVmcz4KICAgIDxtYXJrZXIgaWQ9ImFyciIgdmlld0JveD0iMCAwIDEwIDEwIiByZWZYPSI4IiByZWZZPSI1IiBtYXJrZXJXaWR0aD0iNiIgbWFya2VySGVpZ2h0PSI2IiBvcmllbnQ9ImF1dG8tc3RhcnQtcmV2ZXJzZSI+CiAgICAgIDxwYXRoIGQ9Ik0yIDFMOCA1TDIgOSIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjMzc0MTUxIiBzdHJva2Utd2lkdGg9IjEuNSIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIi8+CiAgICA8L21hcmtlcj4KICA8L2RlZnM+CgogIAogIDxyZWN0IHg9IjMwIiB5PSIxNTIiIHdpZHRoPSIxNCIgaGVpZ2h0PSIxNCIgcng9IjMiIGZpbGw9IiMwNTJlMTYiIHN0cm9rZT0iIzIyYzU1ZSIgc3Ryb2tlLXdpZHRoPSIxLjUiLz4KICA8dGV4dCB4PSI1MiIgeT0iMTYzIiBmb250LXNpemU9IjEyIiBmaWxsPSIjOWNhM2FmIj5DYXB0dXJlZCBieSBzbG93IHF1ZXJ5IGxvZzwvdGV4dD4KICA8cmVjdCB4PSIyNTAiIHk9IjE1MiIgd2lkdGg9IjE0IiBoZWlnaHQ9IjE0IiByeD0iMyIgZmlsbD0iIzNiMGYwZiIgc3Ryb2tlPSIjZWY0NDQ0IiBzdHJva2Utd2lkdGg9IjEiLz4KICA8dGV4dCB4PSIyNzIiIHk9IjE2MyIgZm9udC1zaXplPSIxMiIgZmlsbD0iIzljYTNhZiI+Tm90IGNhcHR1cmVkIC0gbmVlZHMgc2VwYXJhdGUgaW5zdHJ1bWVudGF0aW9uPC90ZXh0PgoKICAKICA8bGluZSB4MT0iMzAiIHkxPSIxODQiIHgyPSI3NzAiIHkyPSIxODQiIHN0cm9rZT0iIzFmMjkzNyIgc3Ryb2tlLXdpZHRoPSIxIi8+CgogIAogIDx0ZXh0IHg9IjQwMCIgeT0iMjA2IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEyIiBmaWxsPSIjNmI3MjgwIj5UeXBpY2FsIDEsMTgwbXMgcmVzcG9uc2UgLSBzbG93IHF1ZXJ5IGxvZyBzZWVzIG9ubHkgODBtczwvdGV4dD4KCiAgCiAgPHJlY3QgeD0iNDAiICB5PSIyMTgiIHdpZHRoPSI3MiIgIGhlaWdodD0iMjgiIHJ4PSI0IiBmaWxsPSIjZWY0NDQ0IiBvcGFjaXR5PSIwLjgiLz4KICA8cmVjdCB4PSIxMTYiIHk9IjIxOCIgd2lkdGg9IjU4IiAgaGVpZ2h0PSIyOCIgcng9IjQiIGZpbGw9IiNlZjQ0NDQiIG9wYWNpdHk9IjAuOCIvPgogIDxyZWN0IHg9IjE3OCIgeT0iMjE4IiB3aWR0aD0iOTYiICBoZWlnaHQ9IjI4IiByeD0iNCIgZmlsbD0iIzIyYzU1ZSIgb3BhY2l0eT0iMC44Ii8+CiAgPHJlY3QgeD0iMjc4IiB5PSIyMTgiIHdpZHRoPSIxMTYiIGhlaWdodD0iMjgiIHJ4PSI0IiBmaWxsPSIjZWY0NDQ0IiBvcGFjaXR5PSIwLjgiLz4KICA8cmVjdCB4PSIzOTgiIHk9IjIxOCIgd2lkdGg9IjcyIiAgaGVpZ2h0PSIyOCIgcng9IjQiIGZpbGw9IiNlZjQ0NDQiIG9wYWNpdHk9IjAuOCIvPgoKICAKICA8dGV4dCB4PSI3NiIgIHk9IjIzNyIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMSIgZmlsbD0iI2ZmZiIgZm9udC13ZWlnaHQ9IjYwMCI+MjUwbXM8L3RleHQ+CiAgPHRleHQgeD0iMTQ1IiB5PSIyMzciIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiNmZmYiIGZvbnQtd2VpZ2h0PSI2MDAiPjIwMG1zPC90ZXh0PgogIDx0ZXh0IHg9IjIyNiIgeT0iMjM3IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjExIiBmaWxsPSIjZmZmIiBmb250LXdlaWdodD0iNjAwIj44MG1zPC90ZXh0PgogIDx0ZXh0IHg9IjMzNiIgeT0iMjM3IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjExIiBmaWxsPSIjZmZmIiBmb250LXdlaWdodD0iNjAwIj40MDBtczwvdGV4dD4KICA8dGV4dCB4PSI0MzQiIHk9IjIzNyIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMSIgZmlsbD0iI2ZmZiIgZm9udC13ZWlnaHQ9IjYwMCI+MjUwbXM8L3RleHQ+CgogIAogIDx0ZXh0IHg9Ijc2IiAgeT0iMjY0IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEwIiBmaWxsPSIjOWNhM2FmIj5Qb29sPC90ZXh0PgogIDx0ZXh0IHg9IjE0NSIgeT0iMjY0IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEwIiBmaWxsPSIjOWNhM2FmIj5Mb2NrPC90ZXh0PgogIDx0ZXh0IHg9IjIyNiIgeT0iMjY0IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEwIiBmaWxsPSIjOWNhM2FmIj5RdWVyeTwvdGV4dD4KICA8dGV4dCB4PSIzMzYiIHk9IjI2NCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMCIgZmlsbD0iIzljYTNhZiI+U2VyaWFsIGF3YWl0czwvdGV4dD4KICA8dGV4dCB4PSI0MzQiIHk9IjI2NCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMCIgZmlsbD0iIzljYTNhZiI+RXh0LiBjYWxsPC90ZXh0PgoKICAKICA8dGV4dCB4PSI1NjAiIHk9IjIzMiIgZm9udC1zaXplPSIxMyIgZm9udC13ZWlnaHQ9IjYwMCIgZmlsbD0iI2Y5ZmFmYiI+VG90YWw6IDEsMTgwbXM8L3RleHQ+CiAgPHRleHQgeD0iNTYwIiB5PSIyNTIiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiM5Y2EzYWYiPkxvZyBzZWVzOiA4MG1zICg3JSk8L3RleHQ+CiAgPHRleHQgeD0iNTYwIiB5PSIyNzAiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiM5Y2EzYWYiPjkzJSBpcyBpbnZpc2libGUgd2l0aG91dDwvdGV4dD4KICA8dGV4dCB4PSI1NjAiIHk9IjI4NSIgZm9udC1zaXplPSIxMSIgZmlsbD0iIzljYTNhZiI+cHJvcGVyIGluc3RydW1lbnRhdGlvbjwvdGV4dD4KPC9zdmc+ ``` Most engineers profile the green box. Production slowdowns live in the red ones. * * * ## Cause 1: Connection Pool Exhaustion Your pool has 10 connections. Under normal load, each request borrows one, runs a query in ~20ms, and returns it. But under load, or when requests hold connections longer than usual, the pool drains. New requests don't fail — they _queue_, waiting silently for a slot to open. With the default `connectionTimeoutMillis: 0` in `node-postgres`, that queue has no timeout. Requests wait indefinitely. The endpoint appears to hang. The database is idle. This is the most common misdiagnosis in backend engineering. It looks exactly like a database problem. ### How to see it ```typescript import { Pool } from 'pg' const pool = new Pool({ max: 10 }) setInterval(() => { console.log({ total: pool.totalCount, // connections created idle: pool.idleCount, // available right now waiting: pool.waitingCount, // requests queued for a connection ← this is your signal }) }, 2000) ``` `waitingCount > 0` under normal load means the pool is the bottleneck. If it's 0 at rest and climbs under traffic, you're undersized or connections are being held too long. ### The fix: configure the pool correctly ```typescript const pool = new Pool({ max: 10, connectionTimeoutMillis: 5_000, // fail after 5s — never leave at 0 idleTimeoutMillis: 30_000, statement_timeout: 10_000, // kill runaway queries that hold connections }) ``` `connectionTimeoutMillis: 5_000` is the most important line. When the pool is exhausted and a request has waited 5 seconds, it gets a clear error instead of hanging forever. That error surfaces as a 503, which is the correct behavior — fail fast, let the client retry. ### The leak pattern A pool that slowly drains over hours (watch `totalCount` dropping without restarts) has a connection leak. The cause is almost always `release()` being skipped on error: ```typescript // LEAKS: if the query throws, release() is never called async function getUser(id: string) { const client = await pool.connect() const result = await client.query('SELECT * FROM users WHERE id = $1', [id]) client.release() // ← skipped when the line above throws return result.rows[0] } // CORRECT: finally() always runs async function getUser(id: string) { const client = await pool.connect() try { const result = await client.query('SELECT * FROM users WHERE id = $1', [id]) return result.rows[0] } finally { client.release() // runs on success, error, and everything in between } } // SIMPLEST: pool.query() manages the lifecycle for you async function getUser(id: string) { const { rows } = await pool.query('SELECT * FROM users WHERE id = $1', [id]) return rows[0] } ``` Use `pool.query()` for single statements. Reach for `pool.connect()` only when you need a transaction or multiple statements on the same connection. ### Pool sizing formula Running multiple service instances? Each one creates its own pool. The math: ```toml max_per_instance = floor((pg_max_connections × 0.8) / num_instances) ``` For 100 `max_connections` and 3 API replicas: `floor(80 / 3) = 26`. Keep 20% headroom because `max_connections` includes superuser connections, replication, and monitoring tools. Above 90%, PostgreSQL starts refusing connections entirely — not gracefully, just hard refusals. At higher scale, run [PgBouncer]() in front of PostgreSQL in transaction pooling mode. It multiplexes hundreds of application connections onto a small pool of actual server connections, letting your instances set `max: 5` while PostgreSQL still only sees 20–25 connections total. * * * ## Cause 2: Queries Blocked on Locks (Not Running Slowly) The slow query log records _execution time_. A query waiting on a lock isn't executing — it's waiting. Those seconds don't appear anywhere in the log, and the query duration from the application's side includes all of them. ```svg PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA4MDAgMjMwIiB3aWR0aD0iMTAwJSIgc3R5bGU9Im1heC13aWR0aDo4MDBweDtkaXNwbGF5OmJsb2NrO21hcmdpbjoxLjVyZW0gMDtmb250LWZhbWlseTpzeXN0ZW0tdWksc2Fucy1zZXJpZiI+CiAgPGRlZnM+CiAgICA8bWFya2VyIGlkPSJibGthcnIiIHZpZXdCb3g9IjAgMCAxMCAxMCIgcmVmWD0iOCIgcmVmWT0iNSIgbWFya2VyV2lkdGg9IjYiIG1hcmtlckhlaWdodD0iNiIgb3JpZW50PSJhdXRvLXN0YXJ0LXJldmVyc2UiPgogICAgICA8cGF0aCBkPSJNMiAxTDggNUwyIDkiIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzRiNTU2MyIgc3Ryb2tlLXdpZHRoPSIxLjUiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIgc3Ryb2tlLWxpbmVqb2luPSJyb3VuZCIvPgogICAgPC9tYXJrZXI+CiAgPC9kZWZzPgoKICA8cmVjdCB3aWR0aD0iODAwIiBoZWlnaHQ9IjIzMCIgZmlsbD0iIzBmMTExNyIgcng9IjEyIi8+CiAgPHRleHQgeD0iNDAwIiB5PSIyOCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMyIgZmlsbD0iIzZiNzI4MCIgZm9udC13ZWlnaHQ9IjUwMCI+SE9XIE9ORSBGT1JHT1RURU4gVFJBTlNBQ1RJT04gTE9DS1MgT1VUIFlPVVIgRU5USVJFIEFQUDwvdGV4dD4KCiAgCiAgPHJlY3QgeD0iMzAiIHk9IjQ4IiB3aWR0aD0iMTkwIiBoZWlnaHQ9IjcyIiByeD0iOCIgZmlsbD0iIzNiMGYwZiIgc3Ryb2tlPSIjZWY0NDQ0IiBzdHJva2Utd2lkdGg9IjEuNSIvPgogIDx0ZXh0IHg9IjEyNSIgeT0iNzQiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTMiIGZvbnQtd2VpZ2h0PSI3MDAiIGZpbGw9IiNmODcxNzEiPkJhdGNoIGpvYjwvdGV4dD4KICA8dGV4dCB4PSIxMjUiIHk9IjkyIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjExIiBmaWxsPSIjZmNhNWE1Ij5pZGxlIGluIHRyYW5zYWN0aW9uPC90ZXh0PgogIDx0ZXh0IHg9IjEyNSIgeT0iMTA4IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjExIiBmaWxsPSIjZmNhNWE1Ij5ob2xkaW5nIGxvY2sgZm9yIDE0IG1pbjwvdGV4dD4KCiAgCiAgPHRleHQgeD0iMTI1IiB5PSIxNDMiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiM2YjcyODAiPmhvbGRzIFJvd0V4Y2x1c2l2ZUxvY2s8L3RleHQ+CiAgPHRleHQgeD0iMTI1IiB5PSIxNTgiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiM2YjcyODAiPm9uIHVzZXJzIHJvdyAjODgyMTwvdGV4dD4KCiAgCiAgPGxpbmUgeDE9IjIyMSIgeTE9Ijg0IiB4Mj0iMjU3IiB5Mj0iODQiIHN0cm9rZT0iIzRiNTU2MyIgc3Ryb2tlLXdpZHRoPSIxLjUiIHN0cm9rZS1kYXNoYXJyYXk9IjUgMyIgbWFya2VyLWVuZD0idXJsKCNibGthcnIpIi8+CiAgPHRleHQgeD0iMjM5IiB5PSI3NiIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMCIgZmlsbD0iIzZiNzI4MCI+YmxvY2tzPC90ZXh0PgoKICAKICA8cmVjdCB4PSIyNTkiIHk9IjU1IiB3aWR0aD0iMTU1IiBoZWlnaHQ9IjU4IiByeD0iOCIgZmlsbD0iIzFjMTkxNyIgc3Ryb2tlPSIjNGI1NTYzIiBzdHJva2Utd2lkdGg9IjEiLz4KICA8dGV4dCB4PSIzMzYiIHk9Ijc5IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEyIiBmb250LXdlaWdodD0iNjAwIiBmaWxsPSIjZDFkNWRiIj5VUERBVEUgdXNlcnM8L3RleHQ+CiAgPHRleHQgeD0iMzM2IiB5PSI5NyIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMSIgZmlsbD0iIzljYTNhZiI+d2FpdGluZyAxMnMuLi48L3RleHQ+CgogIAogIDxsaW5lIHgxPSI0MTQiIHkxPSI4NCIgeDI9IjQ1MCIgeTI9Ijg0IiBzdHJva2U9IiM0YjU1NjMiIHN0cm9rZS13aWR0aD0iMS41IiBzdHJva2UtZGFzaGFycmF5PSI1IDMiIG1hcmtlci1lbmQ9InVybCgjYmxrYXJyKSIvPgogIDx0ZXh0IHg9IjQzMiIgeT0iNzYiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTAiIGZpbGw9IiM2YjcyODAiPmJsb2NrczwvdGV4dD4KCiAgCiAgPHJlY3QgeD0iNDUyIiB5PSI1NSIgd2lkdGg9IjE1NSIgaGVpZ2h0PSI1OCIgcng9IjgiIGZpbGw9IiMxYzE5MTciIHN0cm9rZT0iIzRiNTU2MyIgc3Ryb2tlLXdpZHRoPSIxIi8+CiAgPHRleHQgeD0iNTI5IiB5PSI3OSIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMiIgZm9udC13ZWlnaHQ9IjYwMCIgZmlsbD0iI2QxZDVkYiI+U0VMRUNUICogdXNlcnM8L3RleHQ+CiAgPHRleHQgeD0iNTI5IiB5PSI5NyIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMSIgZmlsbD0iIzljYTNhZiI+d2FpdGluZyA4cy4uLjwvdGV4dD4KCiAgCiAgPHRleHQgeD0iNjI1IiB5PSI3NCIgZm9udC1zaXplPSIxMiIgZmlsbD0iIzZiNzI4MCI+KyA0IG1vcmUgcXVlcmllczwvdGV4dD4KICA8dGV4dCB4PSI2MjUiIHk9IjkyIiBmb250LXNpemU9IjEyIiBmaWxsPSIjNmI3MjgwIj5xdWV1ZWQgYmVoaW5kPC90ZXh0PgoKICAKICA8bGluZSB4MT0iMzAiIHkxPSIxNzUiIHgyPSI3NzAiIHkyPSIxNzUiIHN0cm9rZT0iIzFmMjkzNyIgc3Ryb2tlLXdpZHRoPSIxIi8+CgogIAogIDx0ZXh0IHg9IjgwIiB5PSIxOTciIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSI2MDAiIGZpbGw9IiM5Y2EzYWYiPlRIRSBGSVggIC0+PC90ZXh0PgoKICA8cmVjdCB4PSIxNjAiIHk9IjE4NCIgd2lkdGg9IjI4MCIgaGVpZ2h0PSIzMCIgcng9IjYiIGZpbGw9IiMwNTJlMTYiIHN0cm9rZT0iIzIyYzU1ZSIgc3Ryb2tlLXdpZHRoPSIxIi8+CiAgPHRleHQgeD0iMzAwIiB5PSIyMDQiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiM0YWRlODAiIGZvbnQtZmFtaWx5PSJtb25vc3BhY2UiPmlkbGVfaW5fdHJhbnNhY3Rpb25fc2Vzc2lvbl90aW1lb3V0ID0gMzBzPC90ZXh0PgoKICA8cmVjdCB4PSI0NTQiIHk9IjE4NCIgd2lkdGg9IjE5MCIgaGVpZ2h0PSIzMCIgcng9IjYiIGZpbGw9IiMwNTJlMTYiIHN0cm9rZT0iIzIyYzU1ZSIgc3Ryb2tlLXdpZHRoPSIxIi8+CiAgPHRleHQgeD0iNTQ5IiB5PSIyMDQiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiM0YWRlODAiIGZvbnQtZmFtaWx5PSJtb25vc3BhY2UiPmxvY2tfdGltZW91dCA9IDVzPC90ZXh0Pgo8L3N2Zz4= ``` ### What causes lock chains `idle in transaction` **sessions** — a connection that ran `BEGIN` and never committed holds its row and table locks indefinitely. A long-running query, an idle-in-transaction session, or an ALTER TABLE waiting for its AccessExclusiveLock can each create a chain reaction where everything queues behind them. One forgotten transaction from a batch job or migration script can queue dozens of application requests. **DDL during peak traffic** — `ALTER TABLE` takes an `AccessExclusiveLock` that blocks every read and write on the table. It also _waits_ for current queries to finish before acquiring the lock — meaning it queues behind slow queries, and every new query queues behind it. The result is a complete stall that lasts as long as the migration. ### Find what's blocking what, right now ```sql SELECT blocked.pid AS blocked_pid, left(blocked.query, 80) AS blocked_query, blocking.pid AS blocking_pid, blocking.state AS blocking_state, -- 'idle in transaction' is the red flag left(blocking.query, 80) AS blocking_query, now() - blocking.query_start AS blocking_duration FROM pg_stat_activity blocked JOIN pg_locks blocked_locks ON blocked_locks.pid = blocked.pid AND NOT blocked_locks.granted JOIN pg_locks blocking_locks ON blocking_locks.locktype IS NOT DISTINCT FROM blocked_locks.locktype AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation AND blocking_locks.pid != blocked_locks.pid AND blocking_locks.granted JOIN pg_stat_activity blocking ON blocking.pid = blocking_locks.pid; ``` If `blocking_state` is `idle in transaction` with a duration measured in minutes, that connection forgot to commit. You can terminate it: ```sql SELECT pg_terminate_backend(blocking_pid); ``` ### Lock-wait aggregate view For recurring contention you need to understand over time rather than catch in the moment: ```sql SELECT wait_event_type, wait_event, count(*) AS backends FROM pg_stat_activity WHERE state != 'idle' GROUP BY wait_event_type, wait_event ORDER BY backends DESC; ``` `wait_event_type = 'Lock'` appearing regularly means lock contention is structural, not a one-off. ### Prevention: configure PostgreSQL defensively ```sql -- postgresql.conf -- Kill idle-in-transaction connections after 30 seconds idle_in_transaction_session_timeout = '30s' -- Don't let queries wait on locks indefinitely (especially in migrations) lock_timeout = '5s' -- Log any time a query waits on a lock (near-zero overhead, safe in production) log_lock_waits = on deadlock_timeout = '1s' ``` `idle_in_transaction_session_timeout` is the single most impactful setting most teams aren't running. It makes the category of "forgot to commit" operationally impossible to sustain. For DDL in production, always use `CREATE INDEX CONCURRENTLY` instead of `CREATE INDEX`, and set `lock_timeout` in your migration scripts so they bail out rather than creating a prolonged stall. * * * ## Cause 3: Sequential Awaits (Serialized Independent Operations) `await` is sequential by default. Two `await` calls that don't depend on each other still run one after the other. The total time is their sum. This is the simplest of the five causes and the one that accumulates silently over time as handlers grow. ### The problem ```typescript // Total time: ~300ms (100ms + 100ms + 100ms) const user = await db.users.findById(userId) const orders = await db.orders.recent(userId) const reviews = await db.reviews.byUser(userId) ``` None of these depend on each other. The user query doesn't need to finish for the orders query to start. Running them sequentially is a pure accident of default syntax. ### The fix ```typescript // Total time: ~100ms (slowest of the three) const [user, orders, reviews] = await Promise.all([ db.users.findById(userId), db.orders.recent(userId), db.reviews.byUser(userId), ]) ``` `Promise.all` fires all three concurrently and resolves when the slowest finishes. ### `Promise.all` vs `Promise.allSettled` `Promise.all` rejects the entire batch if any one promise rejects. That's correct when all results are required. For supplementary data (recommendations, notifications, enrichment) where partial failure is acceptable, use `Promise.allSettled`: ```typescript const [profileResult, recsResult, notifResult] = await Promise.allSettled([ db.users.findById(userId), // required recommendations.fetch(userId), // optional — degrades gracefully notifications.unread(userId), // optional — degrades gracefully ]) const user = profileResult.status === 'fulfilled' ? profileResult.value : null const recs = recsResult.status === 'fulfilled' ? recsResult.value : [] const notifications = notifResult.status === 'fulfilled' ? notifResult.value : [] ``` A down recommendations service doesn't take the page with it. ### The large-batch edge case `Promise.all` over a large array hammers your connection pool — 1,000 concurrent queries against a pool of 10 connections will queue 990 of them. Use concurrency limiting for large batches: ```typescript import pLimit from 'p-limit' const limit = pLimit(5) // max 5 concurrent DB queries const results = await Promise.all( userIds.map(id => limit(() => db.users.findById(id))) ) ``` `p-limit` keeps the concurrency bounded regardless of input size. ### How to find these in your codebase Search your handlers for any function that makes more than one database or external call. Any two `await` statements without a data dependency between them are candidates. Profile endpoints and dashboard loaders are the most common offenders — they start with two calls, accumulate to six over a year of feature development, and none of the additions were ever parallelized. * * * ## Cause 4: Event Loop Blocking Node.js runs JavaScript on a single thread. When that thread is occupied by synchronous work — a large sort, a file read, a CPU-heavy parse — every request queued behind it waits. Not in the database. Not in the network. In your process, unable to advance. The signature: uniform latency increase across _all_ endpoints simultaneously. One slow endpoint usually means one slow dependency. All-endpoint degradation means the thread itself is blocked. ### The metric that matters: Event Loop Utilization CPU utilization doesn't catch this well. A process doing CPU-heavy synchronous work can spike ELU to 1.0 while showing moderate CPU — because ELU measures the proportion of time the event loop spends active (doing work) vs. idle (waiting for I/O). A healthy server sits around 0.5–0.7. Sustained above 0.85 means you're out of headroom for bursts and p99 latency degrades. ELU is one of the cleanest saturation signals for Node.js services. It pairs with event loop delay to explain tail latency without guessing — when the loop gets starved by synchronous work, everything queues up: timers fire late, responses wait, and p99 gets ugly. ```typescript import { performance, monitorEventLoopDelay } from 'perf_hooks' let lastELU = performance.eventLoopUtilization() // p99 event loop delay — how long does the worst tick take? const histogram = monitorEventLoopDelay({ resolution: 10 }) histogram.enable() setInterval(() => { const elu = performance.eventLoopUtilization(lastELU) lastELU = performance.eventLoopUtilization() const eldP99 = histogram.percentile(99) / 1e6 // nanoseconds → ms histogram.reset() if (elu.utilization > 0.85) { console.warn(`[loop] ELU=${elu.utilization.toFixed(2)} — saturation risk`) } if (eldP99 > 100) { console.warn(`[loop] p99 delay=${eldP99.toFixed(1)}ms — blocking detected`) } }, 5000) ``` Note: `monitorEventLoopDelay` underreports large spikes — it can't take measurements while the loop is blocked. Use ELU as the primary signal. Note also from the [Node.js source]() that `h.mean` is skewed toward small values over time; use `h.percentile(99)` instead. ### Common blockers **Sync APIs in request handlers:** ```typescript // Blocks the thread for the entire read const config = JSON.parse(fs.readFileSync('./config.json', 'utf8')) // Yields while waiting const raw = await fs.promises.readFile('./config.json', 'utf8') const config = JSON.parse(raw) ``` Any Node.js API with a `Sync` suffix — `fs.readFileSync`, `crypto.pbkdf2Sync`, `zlib.gzipSync` — runs on the main thread. They're fine at startup. They're expensive in request handlers. **In-memory sorts on large result sets:** ```typescript // This sort blocks while sorting — for 50k rows that's measurable const rows = await db.scores.findAll() const sorted = rows.sort((a, b) => b.score - a.score) // The database can sort in microseconds with an index const top100 = await db.query('SELECT * FROM scores ORDER BY score DESC LIMIT 100') ``` Push work into the database when you can. When you can't (complex business logic), move it to a [worker thread]() so it doesn't block the event loop. **Catastrophic regex backtracking:** ```typescript // Works on normal input. On a crafted string, backtracks exponentially. const vulnerable = /^(a+)+$/ vulnerable.test(userInput) // can freeze the thread for seconds ``` Validate and sanitize external inputs before running them through complex patterns. Use [safe-regex]() to audit patterns in your codebase. ### Finding the source in production Measure event loop delays with `node --trace-event-categories v8,node,node.async_hooks`. Pinpoint asynchronous code blocking the main thread for rapid fixes. For deeper profiling, [Clinic.js]() generates flame graphs and event loop analysis without requiring a production deploy: ```bash npx clinic doctor -- node server.js # Run load against it with autocannon or k6 # Generates a flame graph showing exactly where time is spent ``` * * * ## Cause 5: External Calls Without Timeouts or Circuit Breaking Any call to an external service imports that service's reliability and latency into yours. Average latency might be 50ms. At p99 it might be 800ms. During degradation, it might be 30 seconds or infinity. Without a timeout, your thread waits for the full duration. Under load, multiple threads pile up waiting on the same degraded service. The application stalls — not from anything in your code, but from a dependency you didn't guard. ### Always set explicit timeouts ```typescript // axios: timeout in ms const response = await axios.get(url, { timeout: 2_000 }) // native fetch / undici: AbortSignal const response = await fetch(url, { signal: AbortSignal.timeout(2_000), }) ``` No outbound HTTP call should ever be made without a timeout. There is no correct value for "wait forever." ### Circuit breaking for repeated failures A timeout protects individual requests. A circuit breaker protects _all_ requests once a service starts failing repeatedly. The circuit breaker operates as a three-state machine: CLOSED (normal — all calls pass through), OPEN (tripped — all calls immediately return a fallback, no traffic reaches the failing service), and HALF-OPEN (one probe request allowed through; success resets to CLOSED, failure stays OPEN). `opossum` is the standard circuit breaker library for Node.js: ```typescript import CircuitBreaker from 'opossum' const breaker = new CircuitBreaker(externalApi.fetch, { timeout: 2_000, // individual call timeout errorThresholdPercentage: 50, // open after 50% failure rate resetTimeout: 10_000, // try again after 10s volumeThreshold: 5, // need at least 5 calls before tripping }) breaker.fallback(() => null) // return null when circuit is open // Wire events to your observability stack breaker.on('open', () => metrics.increment('circuit.open', { service: 'partner-api' })) breaker.on('close', () => metrics.increment('circuit.close', { service: 'partner-api' })) breaker.on('halfOpen', () => metrics.increment('circuit.halfopen', { service: 'partner-api' })) // Use exactly like the original call const data = await breaker.fire(userId) ``` When the circuit is open, `breaker.fire()` returns the fallback immediately — no network call, no latency, no cascading load. The external service gets time to recover. ```svg PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA4MDAgMjYwIiB3aWR0aD0iMTAwJSIgc3R5bGU9Im1heC13aWR0aDo4MDBweDtkaXNwbGF5OmJsb2NrO21hcmdpbjoxLjVyZW0gMDtmb250LWZhbWlseTpzeXN0ZW0tdWksc2Fucy1zZXJpZiI+CiAgPGRlZnM+CiAgICA8bWFya2VyIGlkPSJjYmFyciIgdmlld0JveD0iMCAwIDEwIDEwIiByZWZYPSI4IiByZWZZPSI1IiBtYXJrZXJXaWR0aD0iNiIgbWFya2VySGVpZ2h0PSI2IiBvcmllbnQ9ImF1dG8tc3RhcnQtcmV2ZXJzZSI+CiAgICAgIDxwYXRoIGQ9Ik0yIDFMOCA1TDIgOSIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjNmI3MjgwIiBzdHJva2Utd2lkdGg9IjEuNSIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIi8+CiAgICA8L21hcmtlcj4KICA8L2RlZnM+CgogIDxyZWN0IHdpZHRoPSI4MDAiIGhlaWdodD0iMjYwIiBmaWxsPSIjMGYxMTE3IiByeD0iMTIiLz4KICA8dGV4dCB4PSI0MDAiIHk9IjI4IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEzIiBmaWxsPSIjNmI3MjgwIiBmb250LXdlaWdodD0iNTAwIj5DSVJDVUlUIEJSRUFLRVIgU1RBVEUgTUFDSElORTwvdGV4dD4KCiAgCiAgPHJlY3QgeD0iNDAiIHk9IjUwIiB3aWR0aD0iMTc1IiBoZWlnaHQ9IjgyIiByeD0iMTAiIGZpbGw9IiMwNTJlMTYiIHN0cm9rZT0iIzIyYzU1ZSIgc3Ryb2tlLXdpZHRoPSIyIi8+CiAgPHRleHQgeD0iMTI3IiB5PSI4MCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxNiIgZm9udC13ZWlnaHQ9IjcwMCIgZmlsbD0iIzRhZGU4MCI+Q0xPU0VEPC90ZXh0PgogIDx0ZXh0IHg9IjEyNyIgeT0iMTAwIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjExIiBmaWxsPSIjODZlZmFjIj5Ob3JtYWwgb3BlcmF0aW9uPC90ZXh0PgogIDx0ZXh0IHg9IjEyNyIgeT0iMTE2IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjExIiBmaWxsPSIjODZlZmFjIj5BbGwgY2FsbHMgcGFzcyB0aHJvdWdoPC90ZXh0PgoKICAKICA8cmVjdCB4PSIzMTIiIHk9IjUwIiB3aWR0aD0iMTc1IiBoZWlnaHQ9IjgyIiByeD0iMTAiIGZpbGw9IiMzYjBmMGYiIHN0cm9rZT0iI2VmNDQ0NCIgc3Ryb2tlLXdpZHRoPSIyIi8+CiAgPHRleHQgeD0iMzk5IiB5PSI4MCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxNiIgZm9udC13ZWlnaHQ9IjcwMCIgZmlsbD0iI2Y4NzE3MSI+T1BFTjwvdGV4dD4KICA8dGV4dCB4PSIzOTkiIHk9IjEwMCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMSIgZmlsbD0iI2ZjYTVhNSI+Q2lyY3VpdCB0cmlwcGVkPC90ZXh0PgogIDx0ZXh0IHg9IjM5OSIgeT0iMTE2IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjExIiBmaWxsPSIjZmNhNWE1Ij5GYWxsYmFjayByZXR1cm5lZCBpbnN0YW50bHk8L3RleHQ+CgogIAogIDxyZWN0IHg9IjU4NCIgeT0iNTAiIHdpZHRoPSIxNzgiIGhlaWdodD0iODIiIHJ4PSIxMCIgZmlsbD0iIzFjMGQwMCIgc3Ryb2tlPSIjZjU5ZTBiIiBzdHJva2Utd2lkdGg9IjIiLz4KICA8dGV4dCB4PSI2NzMiIHk9IjgwIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjE2IiBmb250LXdlaWdodD0iNzAwIiBmaWxsPSIjZmJiZjI0Ij5IQUxGLU9QRU48L3RleHQ+CiAgPHRleHQgeD0iNjczIiB5PSIxMDAiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiNmY2QzNGQiPkNvb2xkb3duIGVsYXBzZWQ8L3RleHQ+CiAgPHRleHQgeD0iNjczIiB5PSIxMTYiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiNmY2QzNGQiPk9uZSBwcm9iZSByZXF1ZXN0IGFsbG93ZWQ8L3RleHQ+CgogIAogIDxsaW5lIHgxPSIyMTUiIHkxPSI3NiIgeDI9IjMxMCIgeTI9Ijc2IiBzdHJva2U9IiM2YjcyODAiIHN0cm9rZS13aWR0aD0iMS41IiBtYXJrZXItZW5kPSJ1cmwoI2NiYXJyKSIvPgogIDx0ZXh0IHg9IjI2MiIgeT0iNjgiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTAiIGZpbGw9IiM5Y2EzYWYiPmZhaWx1cmUgcmF0ZTwvdGV4dD4KICA8dGV4dCB4PSIyNjIiIHk9IjgwIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEwIiBmaWxsPSIjOWNhM2FmIj4+IDUwJTwvdGV4dD4KCiAgCiAgPGxpbmUgeDE9IjQ4NyIgeTE9Ijc2IiB4Mj0iNTgyIiB5Mj0iNzYiIHN0cm9rZT0iIzZiNzI4MCIgc3Ryb2tlLXdpZHRoPSIxLjUiIG1hcmtlci1lbmQ9InVybCgjY2JhcnIpIi8+CiAgPHRleHQgeD0iNTM0IiB5PSI2OCIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMCIgZmlsbD0iIzljYTNhZiI+cmVzZXQgdGltZW91dDwvdGV4dD4KICA8dGV4dCB4PSI1MzQiIHk9IjgwIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEwIiBmaWxsPSIjOWNhM2FmIj5lbGFwc2VkICgxMHMpPC90ZXh0PgoKICAKICA8cGF0aCBkPSJNNjczIDEzMiBRNjczIDE3OCA0MDAgMTc4IFExMjcgMTc4IDEyNyAxMzIiIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzIyYzU1ZSIgc3Ryb2tlLXdpZHRoPSIxLjUiIHN0cm9rZS1kYXNoYXJyYXk9IjUgMyIgbWFya2VyLWVuZD0idXJsKCNjYmFycikiLz4KICA8dGV4dCB4PSI0MDAiIHk9IjE5NiIgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMSIgZmlsbD0iIzRhZGU4MCI+cHJvYmUgc3VjY2VlZHMgIC0gIHJlc2V0IHRvIENMT1NFRDwvdGV4dD4KCiAgCiAgPHBhdGggZD0iTTU4NCAxMDAgUTU0MCAxNDAgNDg5IDEwMCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjZWY0NDQ0IiBzdHJva2Utd2lkdGg9IjEuNSIgc3Ryb2tlLWRhc2hhcnJheT0iNSAzIiBtYXJrZXItZW5kPSJ1cmwoI2NiYXJyKSIvPgogIDx0ZXh0IHg9IjUyNCIgeT0iMTQ4IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIiBmb250LXNpemU9IjEwIiBmaWxsPSIjZjg3MTcxIj5wcm9iZSBmYWlscyAtIHN0YXkgT1BFTjwvdGV4dD4KCiAgCiAgPHRleHQgeD0iNDAwIiB5PSIyMzAiIHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTEiIGZpbGw9IiM0YjU1NjMiPmtleSBjb25maWc6ICBlcnJvclRocmVzaG9sZFBlcmNlbnRhZ2U6IDUwICDCtyAgdGltZW91dDogMjAwMG1zICDCtyAgcmVzZXRUaW1lb3V0OiAxMDAwMG1zICDCtyAgdm9sdW1lVGhyZXNob2xkOiA1PC90ZXh0Pgo8L3N2Zz4= ``` ### The critical path question For every external call in your codebase, ask: _if this service is down, should my endpoint fail?_ If yes — it's in your critical path. Protect it with a tight timeout and alert on failure rate. If no — wrap it in `try/catch`, return a fallback, and don't let its failure mode propagate to the user. ```typescript // Non-critical enrichment: always returns something async function getEnrichment(userId: string) { try { const { data } = await axios.get(`https://partner.api/users/${userId}`, { timeout: 1_500, }) return data } catch { return null // enrichment failed — null is an acceptable degraded response } } // Critical path: let errors propagate so the caller can handle them async function verifyPayment(paymentId: string) { const { data } = await axios.post(`https://payments.api/verify`, { paymentId }, { timeout: 5_000, }) return data // caller gets the error if this fails } ``` * * * ## Cause 0: The Query IS Slow (But The Log Isn't Configured Right) Before concluding the database is innocent, make sure you've actually checked it properly. `log_min_duration_statement` logs queries slower than a threshold. But if it's set to 1000ms (a common default), queries taking 400ms every time — which will destroy a dashboard endpoint — never appear. And `pg_stat_statements` gives you aggregate view of all queries, which surfaces patterns the per-request log can miss. ```sql -- Enable pg_stat_statements (add to shared_preload_libraries + restart) CREATE EXTENSION IF NOT EXISTS pg_stat_statements; -- Queries with highest total execution time SELECT left(query, 100) AS query, calls, round(mean_exec_time::numeric, 2) AS avg_ms, round(total_exec_time::numeric, 2) AS total_ms, round(stddev_exec_time::numeric, 2) AS stddev_ms, -- high stddev = inconsistent rows FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20; ``` `mean_exec_time` shows average query speed. `stddev_exec_time` is the one most engineers miss: high standard deviation means the query is fast most of the time and catastrophically slow occasionally — which is exactly the profile of a query that's fast on warm cache and slow on cold read, or one that behaves well on small result sets and falls apart on large ones. The `wait_event_type` and `wait_event` columns in `pg_stat_activity` show if a query is waiting on a lock, I/O, or another resource — essential for diagnosing why a query appears stuck rather than slow. Configure your postgresql.conf for production visibility: ```conf log_min_duration_statement = 200 # log anything over 200ms — not 1000ms log_lock_waits = on # log whenever a query waits for a lock log_checkpoints = on log_temp_files = 0 # log any temp file creation (sort/hash spills) ``` * * * ## Putting It All Together: The Diagnosis Order When an endpoint is slow and the obvious explanations aren't obvious, run through this in sequence: **1\. Check pool waiting count.** Add the `setInterval` monitor. Is `waitingCount > 0`? If yes: the pool is the bottleneck. Check for leaks or size the pool correctly. **2\. Check for blocked queries.** Run the locking query against `pg_stat_activity`. Any `idle in transaction` connections with duration in the minutes? Kill them and add `idle_in_transaction_session_timeout`. **3\. Check event loop utilization.** Add the ELU monitor. Is it consistently above 0.85? If yes: something is blocking the thread. Use Clinic.js to find it. **4\. Check for sequential awaits.** Read the slow handler. Do two or more `await` calls have no data dependency between them? Move them into `Promise.all`. **5\. Check external call durations.** Every outbound HTTP call should log its response time. A p99 that's 10× the mean means one in a hundred requests is waiting on a degraded service. Add timeouts and circuit breakers. **6\. Check** `pg_stat_statements`**.** Look at `mean_exec_time` and `stddev_exec_time`. High stddev on a frequently-called query is a silent killer. The slow query log is one instrument in a larger set. It catches slow execution — not waiting for connections, not waiting for locks, not blocked threads, not external service degradation. Production latency can live in any of these places, and the only way to find it is to measure all of them. * * * **Further reading:** - [Node.js]() `perf_hooks` [documentation]() — official reference for ELU and event loop delay APIs - [Clinic.js]() — flame graphs and event loop analysis for Node.js - [PgBouncer documentation]() — connection pooler configuration reference - [opossum circuit breaker]() — the standard circuit breaker for Node.js - [p-limit]() — concurrency limiting for `Promise.all` at scale - [pg\_stat\_statements]() — PostgreSQL extension for aggregate query statistics - [PostgreSQL lock monitoring]() — official wiki with locking queries and explanations