All articlesShoaib
Technical deep dive//7 min read

Designing Resumable Uploads for Large Files

Why upload reliability is mostly a coordination problem between the browser, the API, and object storage.

UploadsObject storageReliability
Designing Resumable Uploads for Large Files

The failure mode

A single multipart request treats a large upload as one indivisible event. A mobile connection does not. When the request fails at 90 percent, the system should know what was completed instead of starting from zero.

The design I prefer gives each upload a session, a stable object key, and independently retryable parts. The client can ask the server which parts exist and continue from there.

The important boundary

The API should coordinate intent and authorization; object storage should carry the bytes. Keeping the application server out of the data path reduces memory pressure and makes progress easier to reason about.

Completion still needs a server-side check. A client saying "done" is not enough; the server should verify the parts, size, and content type before publishing the object.

The result

Resumability is not only a performance feature. It changes failure from a frustrating restart into a recoverable state that the product can explain.

The upload lifecycle

The API should create an upload session before any bytes move. That session records the intended filename, size, media type, owner, and expiration. The client then receives short-lived part URLs or a controlled upload endpoint and reports completed parts back to the API.

sequenceDiagram
  participant U as Browser
  participant A as API
  participant O as Object storage
  U->>A: Create upload session
  A-->>U: Session and part instructions
  U->>O: Upload part 1
  U->>O: Upload part 2
  U->>A: Report completed parts
  A->>O: Verify and complete object
  O-->>A: Object metadata
  A-->>U: Published resource

Security and correctness

The session must be scoped to the authenticated user and the intended destination. A client-provided object key should never decide who can read the resulting file. Part URLs should expire, and the server should enforce a maximum size before issuing them.

The completion step is where the system earns trust. Verify that every expected part exists, that the total size is plausible, and that the final object belongs to the session. If processing is required, publish a temporary status first and expose a durable processing state rather than blocking a request until a virus scan or thumbnail job finishes.

async function completeUpload(sessionId: string, parts: UploadedPart[]) {
  const session = await uploads.findActive(sessionId);
  assert(session && session.expiresAt > new Date());
  assert(parts.length === session.expectedParts);
  assert(parts.every((part) => part.size > 0));

  await storage.completeMultipartUpload(session.objectKey, parts);
  return uploads.publish(sessionId);
}

Recovery behavior

Expired sessions need cleanup. Abandoned multipart uploads can otherwise become a quiet storage bill. The UI should let a person resume a recent session, cancel it, or start over when the original file has changed. The server should treat completion as idempotent so a lost response does not cause a second object to be created.