How to Implement Background Jobs & Queues in Small Web Apps

A background job lets a web request hand off work that does not need to finish before the user gets a response. Examples include generating a large export, sending a batch of notifications, or processing a webhook that needs retries.

A queue does not make work reliable by itself. It adds a broker or datastore, worker processes, retry behavior, and another system to monitor. Keep short, user-facing operations in the request when the user needs an immediate result or error. Move work to a queue when it can finish later and you have a plan for failures and status.

What belongs in a background job?

Work	Usually a good fit for a queue?	Why
Validate a form and save a small record	No	The request should return validation and save errors immediately.
Generate a large report or archive	Often	The task may take longer than a useful request can remain open.
Send a webhook or email after a successful change	Often	Delivery can be retried without holding the original request open.
A recurring cleanup or digest	Often	A scheduler can create jobs on a repeatable schedule.
A fast database lookup needed to render a page	Usually not	Adding a queue delays the result and adds infrastructure without helping the user.

Use the queue for work whose completion can be decoupled from the request. If the user needs to see progress, create a durable status record in your application database and return an ID they can check. A queue's completed-job record is not a substitute for application data: queues often remove old jobs to control storage.

A small queue-and-worker example

The example below uses BullMQ with Node.js and Redis. Its producer adds a small report job; a separate worker summarizes the supplied rows and writes the result to a local file. The file write is deterministic, so retrying the same input replaces the same output rather than creating another report.

In a real application, send an identifier such as reportId instead of a large data payload, load the data in the worker, and store the result in storage that every web and worker instance can access. Do not put credentials, private customer data, or filesystem paths from a request into a job payload.

Create the queue in the web process and add a job after validating the request. Use the same Redis connection settings in the worker process:

import { Queue } from "bullmq";

const connection = {
  host: process.env.REDIS_HOST ?? "127.0.0.1",
  port: Number(process.env.REDIS_PORT ?? 6379),
};

const reports = new Queue("reports", {
  connection,
  defaultJobOptions: {
    attempts: 4,
    backoff: { type: "exponential", delay: 1000, jitter: 0.5 },
    removeOnComplete: { age: 3600, count: 1000 },
    removeOnFail: { age: 7 * 24 * 3600 },
  },
});

await reports.add(
  "write-report",
  {
    reportId: "report-123",
    rows: [
      { category: "books", amount: 12 },
      { category: "courses", amount: 8 },
      { category: "books", amount: 5 },
    ],
  },
  { jobId: "report-report-123" },
);

Run the worker as a separate process. A small concurrency setting is a reasonable starting point for I/O work; choose a value that fits your database, external services, and worker memory:

import { mkdir, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { Worker } from "bullmq";

const connection = {
  host: process.env.REDIS_HOST ?? "127.0.0.1",
  port: Number(process.env.REDIS_PORT ?? 6379),
};

const worker = new Worker(
  "reports",
  async (job) => {
    if (job.name !== "write-report" || !Array.isArray(job.data.rows)) {
      throw new Error("Invalid report job");
    }
    if (!/^[a-zA-Z0-9_-]+$/.test(job.data.reportId)) {
      throw new Error("Invalid report ID");
    }

    const totals = Object.create(null);
    for (const row of job.data.rows) {
      if (typeof row.category !== "string" || !Number.isFinite(row.amount)) {
        throw new Error("Invalid report row");
      }
      totals[row.category] = (totals[row.category] ?? 0) + row.amount;
    }

    await mkdir("reports", { recursive: true });
    const outputPath = join("reports", `${job.data.reportId}.json`);
    await writeFile(
      outputPath,
      JSON.stringify({ reportId: job.data.reportId, totals }, null, 2),
      "utf8",
    );
    return { outputPath };
  },
  { connection, concurrency: 2 },
);

worker.on("failed", (job, error) => {
  console.error("Report job failed", {
    jobId: job?.id,
    errorName: error.name,
  });
});

This demonstration uses local files only to keep the worker example small. Local disk is not shared between multiple machines, and the sample does not include a database-backed progress page. For production, put the report in shared object storage, store its status in your application database, and close workers gracefully during deployment. Configure Redis authentication, network access, and TLS for your hosting environment rather than exposing a local-development connection.

The example also retains completed and failed queue records for a bounded period. Choose retention based on how long you need to investigate failures and how much storage you can afford. Do not keep full job payloads indefinitely just to display a user's status.

Make retries safe

A worker can fail after making a change but before the queue records the job as complete. The same job can therefore run again. Design each operation so a retry does not send duplicate mail, charge a card twice, or create multiple exports.

BullMQ supports a maximum number of attempts and fixed or exponential backoff; the example uses four attempts and exponential backoff with jitter. Retry transient errors such as a temporary network failure, but do not endlessly retry invalid input or a permanent authorization failure. Review the BullMQ retry guidance and its idempotent-job pattern before adding side effects.

A stable job ID can suppress duplicate additions while the job remains in the queue. It is not a permanent idempotency guarantee: after a job is removed, the same ID can be added again. For an important operation, record a unique operation or event ID in your database, and use an idempotency key when the external service supports one. Keep each job small enough that partial completion is easy to reason about.

There is also a gap between saving a database change and publishing a queue message. If the web process stops after the database commit but before the enqueue succeeds, the work may never run. When losing that work is unacceptable, write an outbox record in the same database transaction as the business change and run a relay that retries publishing it. AWS's transactional outbox guidance explains this dual-write problem and why consumers still need to handle duplicate messages.

Schedule work and handle poison jobs

Use a delayed job when it should not run before a particular time. Delays are not exact alarms: a busy queue or worker can make processing happen after the target time. For recurring work, BullMQ's current Job Scheduler API can create jobs on an interval or cron schedule. Give each schedule a stable ID so deployment or startup code can upsert it rather than create duplicates.

Some jobs will still fail after their retry limit. Keep enough failure information to investigate, alert on a growing failed count, and provide a deliberate way to correct or replay a job. Avoid automatically replaying a permanently invalid job, which can create a retry loop.

Managed queues may use different delivery mechanics. With Amazon SQS, a received message is hidden for a visibility timeout; if it is not deleted before the timeout expires, it can become available again. Standard queues can deliver a message more than once even during that window, so consumers still need to be idempotent. Set and extend the timeout for long-running work, and use an SQS dead-letter queue with a suitable receive limit to isolate repeated failures. A dead-letter queue needs an alert and an owner; it is not a repair by itself. See AWS's visibility timeout guide for the service-specific behavior.

Give users status and give yourself signals

For work visible in the product, save a status such as queued, running, ready, or failed in your own database. Let the worker update it, and show a useful message or retry action when a job fails. Do not rely on queue history for a user-facing record if your retention policy removes completed jobs.

Monitor the number and age of waiting jobs, active workers, failures, retries, and processing time. Alert when a backlog is older than the time users can reasonably wait, not only when the worker process crashes. Include a job ID in logs so you can trace one operation across the request, worker, database, and external service. For application readiness checks, see our guide to health checks and readiness signals for web applications.

Queues and workers cost more than the code that enqueues a task. Budget for the queue service or datastore, worker compute, storage, network calls, and the extra work caused by retries. Configure retention and concurrency so an unexpected burst does not fill the datastore or overwhelm an email, payment, or database service. Keep the primary application data recoverable too; our guide to database backup and restore plans for small web apps covers that part of the system.

A right-sized starting plan

For a small product, start with one queue, one worker process, bounded concurrency, and one important job type. Persist the user's request and its status, pass an ID to the worker, set a small retry limit with backoff, and make the operation safe to run more than once. Add a dead-letter or failed-job review path and an alert before relying on the queue for critical work.

Skip the queue when the task is already fast, the result must be returned immediately, or you do not yet have a way to monitor and recover failed work. A background worker is useful when it separates a real failure boundary from the web request—not just because the work could run somewhere else.