{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"stop-doing-slow-things-in-request-handlers-background-jobs-with-bullmq-vrcoj","url":"https://zyvop.com/stop-doing-slow-things-in-request-handlers-background-jobs-with-bullmq-vrcoj","title":"Stop Doing Slow Things in Request Handlers: Background Jobs with BullMQ","subtitle":"How to move emails, image processing, webhooks, and slow third-party calls out of your API and into a proper job queue","tldr":"Here is a rule of thumb that will save you a lot of production pain: if it takes more than 200ms, it should not be in your request/response cycle. Sending a wel...","keywords":["background jobs Node.js 2026","job queue Express","BullMQ retry","BullMQ Node.js","BullMQ worker","BullMQ Redis"],"entities":["ZyVOP","background jobs Node.js 2026","job queue Express","BullMQ retry","BullMQ Node.js","BullMQ worker","BullMQ Redis"],"keyTakeaways":["Here is a rule of thumb that will save you a lot of production pain: if it takes more than 200ms, it should not be in your request/response cycle.","Resizing an uploaded image.","Calling a slow third-party API."],"headings":["The Architecture","Setup","Step 1 — The Redis Connection","Step 2 — Define Your Queues","Step 3 — Adding Jobs from Your API","Step 4 — The Worker","Step 5 — Running Workers in Docker","Step 6 — The Dashboard","Step 7 — Idempotency (The Part Everyone Forgets)","Monitoring Queue Health","What to Queue vs What Not To Queue"],"outboundLinks":[],"contentText":"Here is a rule of thumb that will save you a lot of production pain: if it takes more than 200ms, it should not be in your request/response cycle. Sending a welcome email. Resizing an uploaded image. Generating a PDF report. Processing a webhook. Calling a slow third-party API. Every one of these is a candidate for a background job — a task that your API kicks off and immediately returns 200 for, while a separate worker process handles the actual work asynchronously. BullMQ is the standard job queue library for Node.js in 2026. It is built on Redis, handles retries, priorities, scheduled jobs, and failure tracking, and it is what most production Node.js backends are using for this problem. This guide walks through setting it up properly — not just getting it running, but getting it production-ready. The Architecture API Server Redis Worker Process ───────────── ───────────── ───────────────── POST /signup ──add──► [ job queue ] ──pick──► send welcome email └─ return 201 [ job queue ] ──pick──► resize avatar [ job queue ] ──pick──► sync to CRMThe API and the worker are separate processes. In production they can scale independently — if you have an email backlog, you spin up more workers without touching your API servers. Setup npm install bullmq ioredisBullMQ requires Redis 6.2 or higher. Step 1 — The Redis Connection // src/lib/redis.js import Redis from 'ioredis'; // BullMQ requires maxRetriesPerRequest: null // This lets BullMQ manage its own retry logic without ioredis interfering export const bullMQConnection = new Redis({ host: process.env.REDIS_HOST || 'localhost', port: parseInt(process.env.REDIS_PORT || '6379'), password: process.env.REDIS_PASSWORD || undefined, maxRetriesPerRequest: null, // Required for BullMQ enableReadyCheck: false, // Required for BullMQ });Step 2 — Define Your Queues Keep queue definitions in one place so API and worker import the same instances: // src/queues/index.js import { Queue } from 'bullmq'; import { bullMQConnection } from '../lib/redis.js'; const defaultJobOptions = { attempts: 3, // Retry up to 3 times on failure backoff: { type: 'exponential', delay: 1000, // First retry after 1s, then 2s, then 4s }, removeOnComplete: { count: 100 }, // Keep last 100 completed jobs for inspection removeOnFail: { count: 500 }, // Keep last 500 failed jobs for debugging }; export const emailQueue = new Queue('email', { connection: bullMQConnection, defaultJobOptions, }); export const imageQueue = new Queue('image-processing', { connection: bullMQConnection, defaultJobOptions: { ...defaultJobOptions, attempts: 2, // Image processing — only retry twice }, }); export const reportQueue = new Queue('reports', { connection: bullMQConnection, defaultJobOptions: { ...defaultJobOptions, attempts: 1, // Reports are expensive — don't auto-retry }, });Step 3 — Adding Jobs from Your API // src/routes/auth.js import { emailQueue } from '../queues/index.js'; router.post('/signup', async (req, res) => { const user = await createUser(req.body); // Kick off the email job and return immediately // Do NOT await this — you don't want to hold the response await emailQueue.add( 'welcome-email', // Job name — use for filtering in dashboard { userId: user.id, email: user.email, name: user.name, }, { // Override defaults for this specific job priority: 1, // Lower number = higher priority delay: 2000, // Wait 2s before processing (let DB writes settle) } ); res.status(201).json({ id: user.id }); // Email sends in background — user already has their 201 }); // Adding a scheduled/recurring job (runs every day at 8am) import { emailQueue } from '../queues/index.js'; await emailQueue.add( 'daily-digest', { type: 'daily-digest' }, { repeat: { pattern: '0 8 * * *', // Cron syntax tz: 'UTC', }, } );Step 4 — The Worker The worker is a separate process. It connects to the same Redis, picks up jobs, and processes them. // src/workers/email.worker.js import { Worker } from 'bullmq'; import { bullMQConnection } from '../lib/redis.js'; import { sendWelcomeEmail, sendDailyDigest } from '../services/email.js'; const emailWorker = new Worker( 'email', async (job) => { // job.name is the name you passed in add() // job.data is the payload // job.id is the unique job ID // job.attemptsMade is how many times this has been tried console.log(`Processing job ${job.id}: ${job.name} (attempt ${job.attemptsMade + 1})`); switch (job.name) { case 'welcome-email': await sendWelcomeEmail({ to: job.data.email, name: job.data.name, }); // Update job progress (visible in dashboard) await job.updateProgress(100); return { sent: true, recipient: job.data.email }; case 'daily-digest': { const users = await getUsersForDigest(); let sent = 0; for (const user of users) { await sendDailyDigest(user); sent++; // Report progress as percentage await job.updateProgress(Math.round((sent / users.length) * 100)); } return { sent, total: users.length }; } default: throw new Error(`Unknown job name: ${job.name}`); } }, { connection: bullMQConnection, concurrency: 5, // Process up to 5 jobs simultaneously limiter: { max: 10, // Max 10 jobs per duration duration: 1000, // Per second — rate limits your email provider calls }, } ); // Event listeners for observability emailWorker.on('completed', (job, result) => { console.log(`Job ${job.id} completed:`, result); }); emailWorker.on('failed', (job, err) => { console.error(`Job ${job?.id} failed (attempt ${job?.attemptsMade}):`, err.message); // Send to your error tracker (Sentry, etc.) captureError(err, { jobId: job?.id, jobName: job?.name, data: job?.data }); }); emailWorker.on('stalled', (jobId) => { // Job was picked up by a worker that died before completing // BullMQ automatically re-queues stalled jobs console.warn(`Job ${jobId} stalled — will be requeued`); }); // Graceful shutdown — finish current jobs before exiting process.on('SIGTERM', async () => { console.log('Worker shutting down...'); await emailWorker.close(); process.exit(0); });Step 5 — Running Workers in Docker In production, run the worker as a separate container. It shares code with your API but has a different entrypoint. # Dockerfile (shared between API and worker) FROM node:20-alpine WORKDIR /app COPY package*.json ./ RUN npm ci --only=production COPY . .# docker-compose.yml (relevant section) services: api: image: your-app command: node src/server.js # ... worker-email: image: your-app command: node src/workers/email.worker.js restart: unless-stopped depends_on: - redis environment: REDIS_HOST: redis # Scale workers independently during high load: # docker compose up --scale worker-email=5Step 6 — The Dashboard BullMQ has an official UI called Bull Board. It shows queue depths, failed jobs, retry buttons, and job payloads — essential for debugging production issues. npm install @bull-board/express @bull-board/api// src/admin/queues.js import { createBullBoard } from '@bull-board/api'; import { BullMQAdapter } from '@bull-board/api/bullMQAdapter.js'; import { ExpressAdapter } from '@bull-board/express'; import { emailQueue, imageQueue, reportQueue } from '../queues/index.js'; const serverAdapter = new ExpressAdapter(); serverAdapter.setBasePath('/admin/queues'); createBullBoard({ queues: [ new BullMQAdapter(emailQueue), new BullMQAdapter(imageQueue), new BullMQAdapter(reportQueue), ], serverAdapter, }); export default serverAdapter;// Mount it in your app — behind admin auth import queueDashboard from './admin/queues.js'; app.use( '/admin/queues', authenticate, requireRole('admin'), queueDashboard.getRouter() );Step 7 — Idempotency (The Part Everyone Forgets) Jobs can run more than once. A worker can crash mid-job, BullMQ re-queues it, and the job runs again. Your job handlers must be idempotent — running them twice should produce the same result as running them once. For emails, use a deduplication key: await emailQueue.add( 'welcome-email', { userId: user.id, email: user.email }, { jobId: `welcome-${user.id}`, // Unique ID prevents duplicate jobs for the same user } );For database writes, use ON CONFLICT DO NOTHING or check before inserting: // In your worker — safe to run twice await db.query(` INSERT INTO email_logs (user_id, type, sent_at) VALUES ($1, $2, NOW()) ON CONFLICT (user_id, type) DO NOTHING `, [job.data.userId, 'welcome']);Monitoring Queue Health A queue that is growing faster than it is draining is a production incident waiting to happen. Track queue depth as a metric: // Health check endpoint — include queue depths router.get('/health', async (req, res) => { const [emailCounts, imageCounts] = await Promise.all([ emailQueue.getJobCounts(), imageQueue.getJobCounts(), ]); const isHealthy = emailCounts.waiting < 1000 && emailCounts.failed < 50; res.status(isHealthy ? 200 : 503).json({ status: isHealthy ? 'ok' : 'degraded', queues: { email: emailCounts, image: imageCounts, }, }); });getJobCounts() returns { waiting, active, completed, failed, delayed, paused }. If waiting is climbing and active is not increasing, your workers are not keeping up — scale them. What to Queue vs What Not To Queue Queue it: Sending any kind of notification (email, SMS, push) Image or video processing PDF generation Webhooks to third-party services Syncing data to external systems (CRM, analytics) Expensive report generation Anything that calls a rate-limited API Do not queue it: Simple database reads Cache lookups Anything the user is actively waiting for in the same session Sub-10ms operations — the queue overhead costs more than the work The rule is not \"make everything async.\" It is \"make the right things async.\" When you queue the right work, your API becomes dramatically more resilient — a slow email provider no longer means slow signups, and a failed third-party webhook no longer loses data.","contentHash":"sha256:1236d4fae0d87d6fd0606693a366732713ac56541f97681130864ef34858e1f9","authorName":"ZyVOP","authorUrl":"https://zyvop.com/author/zyvop","authorSameAs":["https://zyvop.com","https://github.com/zyvop","https://x.com/zyvop1"],"category":null,"tags":["background jobs Node.js 2026","job queue Express","BullMQ retry","BullMQ Node.js","BullMQ worker","BullMQ Redis"],"audience":"Software engineers and developers building applications with background jobs Node.js 2026","tone":"Instructional, practical, code-first","readingTimeMinutes":6,"wordCount":1349,"faqs":null,"primaryTopic":"background jobs Node.js 2026","publishedAt":"2026-05-25T08:51:16.539Z","updatedAt":"2026-08-23T23:20:00.188Z","canonicalUrl":"https://zyvop.com/stop-doing-slow-things-in-request-handlers-background-jobs-with-bullmq-vrcoj"}