BullMQ 5 + Redis로 Node.js 백그라운드 작업 큐 운영하기

Everyone has experienced what happens when you send an email, call an external payment API, or run a large-scale data transformation inside an API response. BullMQ is a Node.js job queue library backed by Redis — a full TypeScript rewrite of the original Bull library. Now that Bull has entered maintenance mode, BullMQ is the default choice when you need a background job queue in a Node.js stack.

This article covers four patterns — priority queues, delayed execution, retry/backoff, and Dead Letter Queue — with production-ready code for each. Examples target BullMQ 5.x in a TypeScript strict-mode environment.

Core Concepts

Components

If you're new to BullMQ, seeing Queue, Worker, Job, and FlowProducer all at once can be a little confusing. Here's a quick breakdown of each role.

Component	Role
Queue	Entry point for registering jobs. Writes jobs to Redis
Worker	Consumer that pulls jobs from the queue and processes them
Job	Unit of execution. Carries data, priority, delay, and retry settings
FlowProducer	Builds DAG pipelines using parent-child dependencies between jobs
QueueEvents	Subscribes to events such as `completed`, `failed`, and `delayed`

The overall data flow looks like this.

Diagram 1

Installation and Basic Setup

bash

pnpm add bullmq ioredis

typescript

// lib/redis.ts
import { Redis } from 'ioredis';
 
export const redisConnection = new Redis({
  host: process.env.REDIS_HOST ?? 'localhost',
  port: Number(process.env.REDIS_PORT ?? 6379),
  maxRetriesPerRequest: null,
});

Omitting maxRetriesPerRequest: null will cause BullMQ to print a warning. The ioredis default limits the number of retries on command failure, but BullMQ needs to wait indefinitely when the connection drops, so setting this to null is mandatory.

typescript

// queues/email.queue.ts
import { Queue } from 'bullmq';
import { redisConnection } from '../lib/redis';
 
export const emailQueue = new Queue('email', {
  connection: redisConnection,
  defaultJobOptions: {
    removeOnComplete: { count: 1_000, age: 24 * 3600 },
    removeOnFail:     { count: 5_000, age: 7 * 24 * 3600 },
  },
});

Always configure removeOnComplete and removeOnFail. Without them, completed and failed job data will accumulate in Redis indefinitely. The values above — count: 5_000 and age: 7 days — are guidelines for medium-scale services; adjust them to your actual throughput, as high-traffic systems may see Redis memory pressure.

Visualizing Job Flow in Development: Bull Board

When you first set up BullMQ locally, the fastest thing you want to verify is "are jobs actually queuing up and being picked up by the Worker?" Bull Board is the best tool for that.

bash

pnpm add @bull-board/api @bull-board/express

typescript

// app.ts
import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';
import basicAuth from 'express-basic-auth';
import { emailQueue } from './queues/email.queue';
import { paymentQueue } from './queues/payment.queue';
 
const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath('/admin/queues');
 
createBullBoard({
  queues: [
    new BullMQAdapter(emailQueue),
    new BullMQAdapter(paymentQueue),
  ],
  serverAdapter,
});
 
// Always attach an auth middleware in production.
app.use(
  '/admin/queues',
  basicAuth({
    users: { admin: process.env.BULL_BOARD_PASSWORD ?? '' },
    challenge: true,
  }),
  serverAdapter.getRouter(),
);

If you already have Bearer token or session-based auth, use that middleware instead of basicAuth. The key point is that /admin/queues must never be publicly accessible without authentication — anyone who reaches this dashboard can retry or delete failed jobs.

Pattern 1: Priority Queue

BullMQ supports integer priorities from 1 to 2,097,152. Lower numbers are processed first.

typescript

// VIP user transactional email — process immediately
await emailQueue.add('transactional', { to: vipUser.email }, { priority: 1 });
 
// Regular notification email
await emailQueue.add('notification', { to: user.email }, { priority: 5 });
 
// Marketing newsletter — process when idle
await emailQueue.add('newsletter', { to: user.email }, { priority: 10 });

It's natural to wonder "can't I just create three separate queues?" The multi-queue pattern — similar to Sidekiq's queue weight approach — is indeed a valid alternative. Here's how to choose:

A single priority queue is better when: the total job volume isn't large and a single Worker pool handles all job types. Worker configuration stays simple and global ordering is easy to guarantee.
Multiple queues are better when: you need to independently control the pool size or concurrency per priority level. For example, if you want 10 Workers for VIP jobs and 2 for newsletters, attach separate Workers to each queue.

Pattern 2: Delayed Execution

Pass a millisecond value to the delay option. The job is stored in a Redis Sorted Set with its scheduled execution time and automatically moved to the waiting queue once that time has passed.

typescript

// Welcome email 5 minutes after registration
await emailQueue.add(
  'welcome',
  { to: newUser.email, name: newUser.name },
  { delay: 5 * 60 * 1_000 },
);
 
// Receipt 1 hour after payment
await emailQueue.add(
  'receipt',
  { orderId, to: user.email },
  { delay: 60 * 60 * 1_000 },
);
 
// Weekly report every Monday at 9 AM — in Korea Standard Time
await emailQueue.add(
  'weekly-report',
  { type: 'summary' },
  {
    repeat: {
      pattern: '0 9 * * MON',
      tz: 'Asia/Seoul',
    },
  },
);

If you don't specify tz in repeat.pattern, it defaults to UTC. For Korean services, "every Monday at 9 AM" could actually run at 6 PM, so be careful not to omit tz: 'Asia/Seoul'.

Pattern 3: Retry and Backoff

Retrying immediately after an external API call fails just hammers an already-overloaded server. That's why Exponential Backoff exists.

typescript

await paymentQueue.add('confirm-payment', { orderId }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1_000,
    // 1s → 2s → 4s → 8s → 16s
  },
});

The formula is 2^(attemptsMade - 1) × delay(ms).

Some error types make retrying pointless. Errors where retrying won't change the outcome — invalid input values, references to non-existent resources — should be treated as immediate final failures. In BullMQ 5, throw an UnrecoverableError for these cases.

typescript

import { UnrecoverableError } from 'bullmq';
 
async function processor(job: Job) {
  try {
    return await riskyApiCall(job.data);
  } catch (error) {
    if (error instanceof ValidationError) {
      // Retrying won't change the outcome — fail immediately and permanently
      throw new UnrecoverableError(`Validation error: ${error.message}`);
    }
    throw error; // Transient error — trigger retry
  }
}

For finer control, you can compute the delay yourself in the Worker's backoffStrategy. The pattern below adds Jitter (randomized delay) to reduce the Thundering Herd effect when many jobs retry simultaneously.

typescript

import { Worker, Job } from 'bullmq';
 
const worker = new Worker('payment', processor, {
  settings: {
    backoffStrategy: (
      attemptsMade: number,
      _type: string,
      _err: Error,
      _job: Job,
    ): number => {
      const base = Math.min(attemptsMade * 2_000, 30_000);
      const jitter = Math.random() * 1_000;
      return base + jitter;
    },
  },
  connection: redisConnection,
});

In Practice: Payment Confirmation Pipeline

External payment gateway APIs are prone to transient failures. Use exponential backoff to ride out temporary outages, and when retries are exhausted or an unrecoverable error occurs, move the job to a Dead Letter Queue (DLQ) so the on-call team can review it.

Diagram 2

typescript

// queues/payment.queue.ts
import { Queue } from 'bullmq';
import { redisConnection } from '../lib/redis';
 
export interface PaymentJobData {
  orderId: string;
  amount: number;
  userId: string;
}
 
export const paymentQueue = new Queue<PaymentJobData>('payment', {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 2_000 },
    removeOnComplete: { count: 500 },
    removeOnFail: { count: 2_000, age: 7 * 24 * 3600 },
  },
});
 
export const dlqQueue = new Queue('payment-dlq', {
  connection: redisConnection,
});

typescript

// workers/payment.worker.ts
import { Worker, Job, UnrecoverableError } from 'bullmq';
import { PaymentJobData, paymentQueue, dlqQueue } from '../queues/payment.queue';
import { pgApiClient } from '../lib/pg-client';
import { notifyOncall } from '../lib/notifications';
import { redisConnection } from '../lib/redis';
 
async function processPayment(job: Job<PaymentJobData>) {
  const { orderId, amount } = job.data;
 
  await job.updateProgress(10);
 
  try {
    const result = await pgApiClient.confirmPayment({ orderId, amount });
    await job.updateProgress(100);
    return result;
  } catch (error) {
    if (error instanceof InvalidOrderError) {
      // Retrying won't change the outcome — move to DLQ immediately
      throw new UnrecoverableError(`Order error: ${(error as Error).message}`);
    }
    throw error;
  }
}
 
const worker = new Worker<PaymentJobData>('payment', processPayment, {
  connection: redisConnection,
  concurrency: 5,
  stalledInterval: 30_000,  // Check for stalled jobs every 30 seconds
  maxStalledCount: 2,       // Mark as failed after 2 stalls
});

Duplicate payment confirmation jobs for the same orderId can happen in practice. Fixing jobId to the orderId prevents duplicates without overwriting a job already in the queue.

typescript

// Register payment confirmation job — deduplicate by orderId
await paymentQueue.add(
  'confirm-payment',
  { orderId, amount, userId },
  { jobId: orderId },
);

The DLQ handler must cover both exhausted retries and unrecoverable errors.

typescript

worker.on('failed', async (job, error) => {
  if (!job) return;
 
  const maxAttempts = job.opts.attempts ?? 1;
  const isExhausted =
    job.attemptsMade >= maxAttempts || error instanceof UnrecoverableError;
 
  if (isExhausted) {
    try {
      await dlqQueue.add('failed-payment', {
        originalJobId: job.id,
        originalData: job.data,
        error: error.message,
        failedAt: new Date().toISOString(),
      });
      await notifyOncall(`Payment job permanently failed: orderId=${job.data.orderId}`);
    } catch (dlqError) {
      console.error('Failed to enqueue to DLQ:', dlqError);
    }
  }
});
 
// Graceful shutdown — prevent job loss during deployment
process.on('SIGTERM', async () => {
  await worker.close();
  await paymentQueue.close();
  process.exit(0);
});

In Practice: Controlling External API Rate Limits

Control per-second call limits for external APIs like Twilio or Stripe at the Worker level. The key advantage of the limiter option is that global rate limiting is enforced across all Worker instances via Redis, regardless of how many are running.

typescript

const twilioWorker = new Worker('sms', async (job: Job) => {
  const { to, body } = job.data;
  await twilioClient.messages.create({
    to,
    body,
    from: process.env.TWILIO_FROM,
  });
}, {
  connection: redisConnection,
  concurrency: 10,
  limiter: {
    max: 100,        // Maximum 100 jobs
    duration: 1_000, // Per 1 second
  },
});

In Practice: Building a Data Pipeline with FlowProducer

For step-by-step pipelines with sequential dependencies — fetch → transform → store — FlowProducer is the right fit. Parent jobs only execute after all their children have completed.

The JSON tree passed to FlowProducer is defined in reverse, with the top-level node being the parent, but actual execution flows bottom-up (children first, then parent).

Diagram 3

typescript

// Register pipeline with FlowProducer
import { FlowProducer } from 'bullmq';
 
const flow = new FlowProducer({ connection: redisConnection });
 
await flow.add({
  name: 'store-result',      // Parent — runs last
  queueName: 'pipeline',
  data: { step: 'store' },
  children: [
    {
      name: 'transform-data',
      queueName: 'pipeline',
      data: { step: 'transform' },
      children: [
        {
          name: 'fetch-data', // Deepest child — runs first
          queueName: 'pipeline',
          data: { step: 'fetch', source: 'api' },
        },
      ],
    },
  ],
});

A child job's return value is accessible in the parent via job.getChildrenValues(). This lets you pass data between pipeline stages without needing separate storage.

typescript

// workers/pipeline.worker.ts
import { Worker, Job } from 'bullmq';
import { redisConnection } from '../lib/redis';
 
const pipelineWorker = new Worker('pipeline', async (job: Job) => {
  switch (job.name) {
    case 'fetch-data': {
      const raw = await fetchFromApi(job.data.source);
      return raw; // Parent accesses this via getChildrenValues()
    }
    case 'transform-data': {
      const childValues = await job.getChildrenValues();
      const rawData = Object.values(childValues)[0];
      return transform(rawData);
    }
    case 'store-result': {
      const childValues = await job.getChildrenValues();
      const transformed = Object.values(childValues)[0];
      await saveToDatabase(transformed);
      break;
    }
  }
}, { connection: redisConnection });

getChildrenValues() returns an object of the form { 'pipeline:jobId': returnValue }. When there is only one child, Object.values(childValues)[0] is the simplest way to extract the value.

Common Mistakes

Omitting removeOnComplete/removeOnFail Completed and failed job data will accumulate in Redis forever. Always set these in defaultJobOptions.

Applying retries to permanent errors Putting attempts: 10 on errors that won't change with retrying — such as invalid input or non-existent IDs — only generates meaningless load. Classify your errors: retry only transient ones, and use UnrecoverableError to fail permanent errors immediately.

No graceful shutdown Deploying without a SIGTERM handler can leave in-flight jobs as stalled, causing them to run twice. Always include a worker.close() call.

Missing try-catch in async event handlers Exceptions inside worker.on('failed', async () => { ... }) become unhandled rejections. DLQ enqueuing and notification sending must be wrapped.

No timezone specified in repeat.pattern The default is UTC. For services targeting a specific timezone, always include tz (e.g., tz: 'Asia/Seoul').

Not using jobId for deduplication In domains where the same job can be registered more than once (payment confirmation, notification sending, etc.), pin jobId to a business identifier to prevent duplicates.

When to Choose BullMQ and When to Consider Alternatives

Advantages

Item	Description
TypeScript-first	Job data types are expressed as generics, validated at compile time
Simple infrastructure	Queue runs on Redis alone — no separate message broker needed
Rich feature set	Priority, delay, repeat, rate limiting, and flows all supported
Stall protection	Jobs are automatically returned to the waiting queue if a Worker crashes
OpenTelemetry support	Officially supported from late 5.x. End-to-end tracing with Datadog and Grafana Tempo

Caveats

Item	Description
Redis availability dependency	If Redis goes down, the entire queue stops. Sentinel or Cluster is recommended for production
No built-in DLQ	Must be implemented manually using the `worker.on('failed', ...)` pattern
Priority overhead	Using a priority queue incurs additional Redis operations. Benchmark if you need extreme throughput
Cross-language limitations	If Python or Go needs to consume the same queue, RabbitMQ or Kafka is a better fit

Technology Selection Guide

	BullMQ	RabbitMQ	Kafka	AWS SQS
Primary language	Node.js-centric	Polyglot	Polyglot	Polyglot
Throughput	Sufficiently high	High	Very high	Managed
Setup complexity	Low	Medium	High	Low
Best for	Node.js job processing	Cross-language messaging	Event streaming	AWS infrastructure

Production Checklist

Once you've verified behavior in your development environment, check the following items before deploying to production.

References

#BullMQ#Redis#Node.js#TypeScript#Job Queue#Worker

Backend

A Rust API Server That Catches SQL Errors at Build Time — Axum + SQLx Type-Safe Patterns Explained

If you've worked with Go or Node.js backends long enough, you've probably experienced this at least once: a schema change causes a missed query update and a 500 error blows up in production, or a bad type cast hides an error until runtime. "High test coverage will save us," you think — but in practice, edge-case queries have a way of slipping through the tests.

July 12, 202622 min read

Core Concepts

Components

Installation and Basic Setup

Visualizing Job Flow in Development: Bull Board

Pattern 1: Priority Queue

Pattern 2: Delayed Execution

Pattern 3: Retry and Backoff

In Practice: Payment Confirmation Pipeline

In Practice: Controlling External API Rate Limits

In Practice: Building a Data Pipeline with FlowProducer

Common Mistakes

When to Choose BullMQ and When to Consider Alternatives

Advantages

Caveats

Technology Selection Guide

Production Checklist

References

Core Concepts

Components

Installation and Basic Setup

Visualizing Job Flow in Development: Bull Board

Pattern 1: Priority Queue

Pattern 2: Delayed Execution

Pattern 3: Retry and Backoff

In Practice: Payment Confirmation Pipeline

In Practice: Controlling External API Rate Limits

In Practice: Building a Data Pipeline with FlowProducer

Common Mistakes

When to Choose BullMQ and When to Consider Alternatives

Advantages

Caveats

Technology Selection Guide

Production Checklist

References

Recommended Posts

Bun 1.2 Built-in SQLite for a Zero-Dependency Local-First Backend

ElysiaJS 1.0 in Practice: Building a Type-Safe REST API with Bun's Native Framework

Building a Production Job Queue with BullMQ — Implementing Priority, Retry Backoff, and Dead Letter Queue in TypeScript

Elysia.js + Bun으로 서버-클라이언트 타입을 한 번에 묶는 법

A Rust API Server That Catches SQL Errors at Build Time — Axum + SQLx Type-Safe Patterns Explained

How to Encode Error Information in Types for TypeScript Backends: A Practical Guide to Effect-TS