Retry Logic Is Not Enough for Reliable Background Jobs

The naive loop

Take a job, catch an error, wait, and try again. It looks reasonable until a permanent validation error burns through retries, or a slow dependency causes every worker to pile up at once.

A production job needs attempt metadata, a backoff policy with jitter, a maximum age, and a dead-letter state that a person can inspect.

Classify before retrying

Timeouts and temporary capacity failures are often retryable. Invalid input, missing permissions, and a rejected business rule usually are not. The worker should make that distinction explicit rather than treating every exception as transient.

The operator experience

A job is not reliable if the only way to understand it is to search logs. Show what is waiting, what failed, why it failed, and what action will happen next.

A job state machine

The queue is only transport. The durable job record is what lets the product explain state across worker restarts and browser refreshes. I would keep attempts, next retry time, last error, and a lease owner next to the job status.

stateDiagram-v2
  [*] --> Queued
  Queued --> Running: worker claims lease
  Running --> Succeeded: effect committed
  Running --> RetryableFailure: temporary error
  RetryableFailure --> Queued: backoff expires
  Running --> DeadLettered: permanent error or max attempts
  DeadLettered --> Queued: operator retries

Leases and duplicates

A worker can crash after completing an external side effect but before acknowledging the queue. The job may run again. That is why the effect itself needs an idempotency key or a reconciliation step; a worker lease alone cannot guarantee exactly-once behavior.

const job = await jobs.claim({
  leaseDurationMs: 30_000,
  workerId,
});

if (job) {
  try {
    await sendInvoice({ idempotencyKey: job.id, invoiceId: job.input.invoiceId });
    await jobs.succeed(job.id);
  } catch (error) {
    await jobs.fail(job.id, classify(error));
  }
}

The worker should renew a lease for long jobs, but the final write must still verify ownership. Otherwise a slow worker can finish after another worker has already reclaimed the job.

The operator experience

A job is not reliable if the only way to understand it is to search logs. Show what is waiting, what failed, why it failed, and what action will happen next. Manual retry should be explicit because retrying a non-idempotent external action can be worse than leaving it failed.