A stream is a promise
When a system says it is real time, users infer more than low latency. They expect updates to be coherent, duplicates to be harmless, and a brief outage not to erase the story of what happened.
That means every pipeline needs an explicit position: an offset, sequence number, or event timestamp that can be used to resume and diagnose gaps.
Backpressure is product behavior
When consumers fall behind, silently buffering forever is not a strategy. The system needs a policy: slow producers, shed low-value updates, or switch to a compact snapshot. Each choice should be visible in the product where it affects trust.
I would rather show a slightly older, complete view than a rapidly changing view with unknown gaps.
The operational payoff
Once replay and lag are first-class concepts, debugging becomes a timeline exercise instead of guesswork.
Separate the three timelines
There is event time, processing time, and presentation time. Event time is when the source says something happened. Processing time is when the pipeline accepted and transformed it. Presentation time is when a consumer made it visible. A healthy design keeps those timestamps instead of pretending they are interchangeable.
flowchart LR
A[Source event time] --> B[Ingestion offset]
B --> C[Validation and enrichment]
C --> D[Durable stream]
D --> E[Consumer checkpoint]
E --> F[Read model]
F --> G[Displayed state]
Backpressure policies
When a consumer is slower than its producer, the pipeline needs a deliberate response. Bounded queues protect memory. Checkpoints protect progress. A compact snapshot can replace a backlog of low-value intermediate updates. The right policy depends on whether the consumer needs every event for audit or only the newest state for display.
I would measure lag as a first-class product signal and alert on its trend rather than one noisy sample. A dashboard that says "live" while its consumer is minutes behind creates a more dangerous illusion than a dashboard that openly says "delayed."
async function consume(batch: Event[]) {
for (const event of batch) {
if (await checkpoints.has(event.stream, event.offset)) continue;
await applyEvent(event);
await checkpoints.save(event.stream, event.offset);
}
}
Replays and schema changes
Replaying an old event through new code can produce a different result. Version event schemas and keep transformations explicit. A replay tool should support a bounded time range, a dry run, and comparison against the current read model before it writes anything.
The ability to replay is valuable only when it is safe enough for an operator to use during an incident.