Observability for Backend Systems: Logs, Metrics, and Traces

Start with questions

Can we tell which customer actions are failing? Did latency increase at ingestion, storage, or rendering? Are retries hiding a dependency outage? These questions should shape instrumentation before dashboards do.

Logs explain individual decisions. Metrics reveal change over time. Traces connect work across service boundaries. None of them is sufficient alone.

Add context that travels

A request ID is useful, but a business operation ID is better. Carry the identity of the workflow through API calls, jobs, database writes, and emitted events. That lets an incident be reconstructed from the user perspective.

The best signal is the one that shortens the path from symptom to responsible boundary.

Instrument the workflow, not just the server

For an order, upload, or AI action, the useful trace begins when the user submits the intent and ends when the durable outcome is visible. Record the operation name, actor type, resource ID, dependency timings, retry count, and final state. Avoid putting secrets or full personal data into logs just because it is convenient during development.

flowchart LR
  A[User operation] --> B[Trace span]
  B --> C[API metrics]
  B --> D[Database span]
  B --> E[Queue and worker span]
  B --> F[Structured application log]
  C --> G[Operational dashboard]
  D --> G
  E --> G
  F --> H[Incident investigation]

Useful measurements

Latency percentiles reveal tail behavior that averages hide. Error rate should be split by operation and dependency. Queue lag should be measured by age, not only count. A cache hit rate is useful only when paired with the cost of misses and the correctness policy for stale data.

const span = tracer.startSpan("invoice.issue");
span.setAttribute("invoice.id", invoiceId);
span.setAttribute("request.id", requestId);

try {
  const result = await issueInvoice(invoiceId);
  span.setAttribute("outcome", "success");
  return result;
} catch (error) {
  span.recordException(error);
  span.setAttribute("outcome", "failure");
  throw error;
} finally {
  span.end();
}

The goal is not to collect every possible signal. It is to make the most important user journeys diagnosable without reconstructing them from unrelated log lines.