Kafka incidents are rarely explained by one lag graph. A trustworthy investigation follows a business event from producer intent through broker acceptance, partition storage, consumer ownership, durable effect, and user-visible outcome. Each layer proves something different.
Quick answer
Begin with affected user outcomes and event cohorts. Separate producer failures, broker or partition health, consumer lag, processing errors, downstream saturation, and incorrect business state. Mitigate with reversible bounded actions, preserve event identity and order constraints, and verify recovery through business reconciliation—not only falling lag.
Prerequisites
Read Observability for Backend Systems, Kafka Topics, Partitions, Consumer Groups, and Ordering, and Reliable Spring Kafka Consumers.
Establish user impact
Define the symptom before changing the cluster: orders remain pending, inventory projections are stale, notifications are delayed, or duplicate charges are reported. Record the start time, tenants, event types, keys, regions, producer versions, consumer groups, and business SLO. Lag without user impact may be expected batch delay; zero lag can coexist with lost or incorrectly acknowledged work.
Select a small set of stable event IDs and aggregate IDs. Trace them through protected producer logs, broker metadata, consumer diagnostics, deduplication records, and business tables. Do not paste sensitive payloads into incident chat or use customer values as metric labels.
Separate the evidence layers
Producer evidence includes outbox age, send attempts, acknowledgment failures, serialization errors, and producer throttling. Broker evidence includes controller and broker availability, under-replicated or offline partitions, ISR changes, request latency, disk and network saturation, and quota effects. Consumer evidence includes assigned partitions, poll health, rebalance activity, lag by partition, processing duration, retry/DLT movement, and offset commits.
Business evidence includes the durable projection version, idempotency disposition, external provider reference, and reconciliation status. A committed offset is not business evidence. A successful database update is not proof that an external provider completed.
Diagnose skew before scaling
Average lag hides a hot partition. Compare maximum and percentile lag with per-partition arrival and processing rates. A hot key, poison record, slow downstream dependency, or one stuck consumer can dominate one partition while the group appears healthy overall.
Adding consumers cannot improve traditional consumer-group parallelism beyond available partitions. Adding partitions during an incident changes routing and may affect key-based order. Scale only after proving the current constraint and preserving the correctness contract.
Retry storms and recovery load
When a dependency slows, retries increase offered load and retain records longer. Non-blocking retry topics can move failures aside but may reorder state. Blocking retry can stall a partition. Bound attempts, apply jitter where appropriate, stop permanent failures, and protect the downstream service with concurrency limits.
During recovery, a large backlog can overload the database or external API even after Kafka is healthy. Ramp consumers deliberately, watch queue age and business latency, and prefer useful-work completion over clearing lag as fast as possible.
Failure scenario
Order projection lag rises after a deployment. Broker health is normal, but one consumer repeatedly deserializes a new enum and sends it through retry topics. Later versions process first and older retries overwrite them. The correct mitigation is to stop the incompatible consumer path, prevent stale writes, deploy a compatible reader, and replay the quarantined cohort under version guards—not to add consumers blindly.
Reversible mitigation
Pause an affected partition or event type when continued processing causes harm. Reduce concurrency when a database is saturated. Disable nonessential consumers independently. Quarantine permanent failures, preserve offsets and original event IDs, and document every manual seek or replay. Avoid deleting topics, resetting entire groups, or expanding partitions without a reviewed recovery plan.
Common mistakes
- Declaring recovery when lag reaches zero but business state remains wrong.
- Resetting offsets without preserving which effects already committed.
- Scaling every consumer while the downstream database is the bottleneck.
- Replaying a DLT without idempotency, authorization, or rate limits.
- Treating liveness, readiness, and broker availability as the same signal.
Production validation
Verify new events complete within the SLO, the backlog age decreases sustainably, retries and DLT arrivals return to baseline, and downstream saturation remains bounded. Reconcile a sampled and aggregate set of source events against projections and external references. Confirm no stale version overwrote newer state and no manual seek created a gap. Preserve the timeline, hypotheses, mitigations, event cohorts, and follow-up owners for the post-incident review.
Sources
- Apache Kafka documentation, accessed 2026-08-18: https://kafka.apache.org/documentation/.
- Apache Kafka 4.1 design documentation, accessed 2026-08-18: https://kafka.apache.org/41/design/design/.
- Spring for Apache Kafka monitoring and listener reference, accessed 2026-08-18: https://docs.spring.io/spring-kafka/reference/kafka.html.
Related reading
Follow the Event-Driven Systems Learning Path and topic cluster. Use Production Incident Troubleshooting with Logs, Metrics, and Traces, Dead Letter Queue Explained, and Event Ordering and State Convergence during recovery. The broader course remains available under System Design.