Stop Doing Slow Things in Request Handlers: Background Jobs with BullMQ

Here is a rule of thumb that will save you a lot of production pain: if it takes more than 200ms, it should not be in your request/response cycle.

Sending a welcome email. Resizing an uploaded image. Generating a PDF report. Processing a webhook. Calling a slow third-party API. Every one of these is a candidate for a background job — a task that your API kicks off and immediately returns 200 for, while a separate worker process handles the actual work asynchronously.

BullMQ is the standard job queue library for Node.js in 2026. It is built on Redis, handles retries, priorities, scheduled jobs, and failure tracking, and it is what most production Node.js backends are using for this problem. This guide walks through setting it up properly — not just getting it running, but getting it production-ready.

The Architecture

API Server                    Redis                      Worker Process
─────────────                 ─────────────              ─────────────────
POST /signup        ──add──►  [ job queue ]  ──pick──►   send welcome email
  └─ return 201               [ job queue ]  ──pick──►   resize avatar
                              [ job queue ]  ──pick──►   sync to CRM

The API and the worker are separate processes. In production they can scale independently — if you have an email backlog, you spin up more workers without touching your API servers.

Setup

npm install bullmq ioredis

BullMQ requires Redis 6.2 or higher.

Step 1 — The Redis Connection

// src/lib/redis.js
import Redis from 'ioredis';

// BullMQ requires maxRetriesPerRequest: null
// This lets BullMQ manage its own retry logic without ioredis interfering
export const bullMQConnection = new Redis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT || '6379'),
  password: process.env.REDIS_PASSWORD || undefined,
  maxRetriesPerRequest: null,   // Required for BullMQ
  enableReadyCheck: false,      // Required for BullMQ
});

Step 2 — Define Your Queues

Keep queue definitions in one place so API and worker import the same instances:

// src/queues/index.js
import { Queue } from 'bullmq';
import { bullMQConnection } from '../lib/redis.js';

const defaultJobOptions = {
  attempts: 3,                          // Retry up to 3 times on failure
  backoff: {
    type: 'exponential',
    delay: 1000,                        // First retry after 1s, then 2s, then 4s
  },
  removeOnComplete: { count: 100 },     // Keep last 100 completed jobs for inspection
  removeOnFail: { count: 500 },         // Keep last 500 failed jobs for debugging
};

export const emailQueue = new Queue('email', {
  connection: bullMQConnection,
  defaultJobOptions,
});

export const imageQueue = new Queue('image-processing', {
  connection: bullMQConnection,
  defaultJobOptions: {
    ...defaultJobOptions,
    attempts: 2,                        // Image processing — only retry twice
  },
});

export const reportQueue = new Queue('reports', {
  connection: bullMQConnection,
  defaultJobOptions: {
    ...defaultJobOptions,
    attempts: 1,                        // Reports are expensive — don't auto-retry
  },
});

Step 3 — Adding Jobs from Your API

// src/routes/auth.js
import { emailQueue } from '../queues/index.js';

router.post('/signup', async (req, res) => {
  const user = await createUser(req.body);

  // Kick off the email job and return immediately
  // Do NOT await this — you don't want to hold the response
  await emailQueue.add(
    'welcome-email',                  // Job name — use for filtering in dashboard
    {
      userId: user.id,
      email: user.email,
      name: user.name,
    },
    {
      // Override defaults for this specific job
      priority: 1,                    // Lower number = higher priority
      delay: 2000,                    // Wait 2s before processing (let DB writes settle)
    }
  );

  res.status(201).json({ id: user.id });
  // Email sends in background — user already has their 201
});

// Adding a scheduled/recurring job (runs every day at 8am)
import { emailQueue } from '../queues/index.js';

await emailQueue.add(
  'daily-digest',
  { type: 'daily-digest' },
  {
    repeat: {
      pattern: '0 8 * * *',          // Cron syntax
      tz: 'UTC',
    },
  }
);

Step 4 — The Worker

The worker is a separate process. It connects to the same Redis, picks up jobs, and processes them.

// src/workers/email.worker.js
import { Worker } from 'bullmq';
import { bullMQConnection } from '../lib/redis.js';
import { sendWelcomeEmail, sendDailyDigest } from '../services/email.js';

const emailWorker = new Worker(
  'email',
  async (job) => {
    // job.name is the name you passed in add()
    // job.data is the payload
    // job.id is the unique job ID
    // job.attemptsMade is how many times this has been tried

    console.log(`Processing job ${job.id}: ${job.name} (attempt ${job.attemptsMade + 1})`);

    switch (job.name) {
      case 'welcome-email':
        await sendWelcomeEmail({
          to: job.data.email,
          name: job.data.name,
        });
        // Update job progress (visible in dashboard)
        await job.updateProgress(100);
        return { sent: true, recipient: job.data.email };

      case 'daily-digest': {
        const users = await getUsersForDigest();
        let sent = 0;
        for (const user of users) {
          await sendDailyDigest(user);
          sent++;
          // Report progress as percentage
          await job.updateProgress(Math.round((sent / users.length) * 100));
        }
        return { sent, total: users.length };
      }

      default:
        throw new Error(`Unknown job name: ${job.name}`);
    }
  },
  {
    connection: bullMQConnection,
    concurrency: 5,               // Process up to 5 jobs simultaneously
    limiter: {
      max: 10,                    // Max 10 jobs per duration
      duration: 1000,             // Per second — rate limits your email provider calls
    },
  }
);

// Event listeners for observability
emailWorker.on('completed', (job, result) => {
  console.log(`Job ${job.id} completed:`, result);
});

emailWorker.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed (attempt ${job?.attemptsMade}):`, err.message);
  // Send to your error tracker (Sentry, etc.)
  captureError(err, { jobId: job?.id, jobName: job?.name, data: job?.data });
});

emailWorker.on('stalled', (jobId) => {
  // Job was picked up by a worker that died before completing
  // BullMQ automatically re-queues stalled jobs
  console.warn(`Job ${jobId} stalled — will be requeued`);
});

// Graceful shutdown — finish current jobs before exiting
process.on('SIGTERM', async () => {
  console.log('Worker shutting down...');
  await emailWorker.close();
  process.exit(0);
});

Step 5 — Running Workers in Docker

In production, run the worker as a separate container. It shares code with your API but has a different entrypoint.

# Dockerfile (shared between API and worker)
FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .

# docker-compose.yml (relevant section)
services:
  api:
    image: your-app
    command: node src/server.js
    # ...

  worker-email:
    image: your-app
    command: node src/workers/email.worker.js
    restart: unless-stopped
    depends_on:
      - redis
    environment:
      REDIS_HOST: redis
    # Scale workers independently during high load:
    # docker compose up --scale worker-email=5

Step 6 — The Dashboard

BullMQ has an official UI called Bull Board. It shows queue depths, failed jobs, retry buttons, and job payloads — essential for debugging production issues.

npm install @bull-board/express @bull-board/api

// src/admin/queues.js
import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter.js';
import { ExpressAdapter } from '@bull-board/express';
import { emailQueue, imageQueue, reportQueue } from '../queues/index.js';

const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath('/admin/queues');

createBullBoard({
  queues: [
    new BullMQAdapter(emailQueue),
    new BullMQAdapter(imageQueue),
    new BullMQAdapter(reportQueue),
  ],
  serverAdapter,
});

export default serverAdapter;

// Mount it in your app — behind admin auth
import queueDashboard from './admin/queues.js';

app.use(
  '/admin/queues',
  authenticate,
  requireRole('admin'),
  queueDashboard.getRouter()
);

Step 7 — Idempotency (The Part Everyone Forgets)

Jobs can run more than once. A worker can crash mid-job, BullMQ re-queues it, and the job runs again. Your job handlers must be idempotent — running them twice should produce the same result as running them once.

For emails, use a deduplication key:

await emailQueue.add(
  'welcome-email',
  { userId: user.id, email: user.email },
  {
    jobId: `welcome-${user.id}`,   // Unique ID prevents duplicate jobs for the same user
  }
);

For database writes, use ON CONFLICT DO NOTHING or check before inserting:

// In your worker — safe to run twice
await db.query(`
  INSERT INTO email_logs (user_id, type, sent_at)
  VALUES ($1, $2, NOW())
  ON CONFLICT (user_id, type) DO NOTHING
`, [job.data.userId, 'welcome']);

Monitoring Queue Health

A queue that is growing faster than it is draining is a production incident waiting to happen. Track queue depth as a metric:

// Health check endpoint — include queue depths
router.get('/health', async (req, res) => {
  const [emailCounts, imageCounts] = await Promise.all([
    emailQueue.getJobCounts(),
    imageQueue.getJobCounts(),
  ]);

  const isHealthy =
    emailCounts.waiting < 1000 &&
    emailCounts.failed < 50;

  res.status(isHealthy ? 200 : 503).json({
    status: isHealthy ? 'ok' : 'degraded',
    queues: {
      email: emailCounts,
      image: imageCounts,
    },
  });
});

getJobCounts() returns { waiting, active, completed, failed, delayed, paused }. If waiting is climbing and active is not increasing, your workers are not keeping up — scale them.

What to Queue vs What Not To Queue

Queue it:

• Sending any kind of notification (email, SMS, push)

• Image or video processing

• PDF generation

• Webhooks to third-party services

• Syncing data to external systems (CRM, analytics)

• Expensive report generation

• Anything that calls a rate-limited API

Do not queue it:

• Simple database reads

• Cache lookups

• Anything the user is actively waiting for in the same session

• Sub-10ms operations — the queue overhead costs more than the work

The rule is not "make everything async." It is "make the right things async." When you queue the right work, your API becomes dramatically more resilient — a slow email provider no longer means slow signups, and a failed third-party webhook no longer loses data.