Kafka stores records in ordered partition logs. Topics organize records, partitions provide storage and parallelism, consumer groups divide partition ownership, and offsets identify positions. Those pieces create useful guarantees, but only within explicit boundaries.
Quick answer
Kafka preserves log order within one partition, not across an entire multi-partition topic. A traditional consumer group assigns a partition to one consumer at a time, while one consumer may own several partitions. Choose a key from the business entity that needs order, then make consumers safe for retries, rebalances, and out-of-order business updates.
Prerequisites
Start with Event-Driven Architecture Decisions and Message Queues Explained. You should already understand that broker delivery and business completion are different outcomes.
Topics and partitions
A topic is a named stream. Each partition is an append-only sequence with monotonically increasing offsets. Replication protects partition data according to Kafka’s broker and producer configuration, but an offset is only a position. It does not prove that a consumer’s transaction committed or that an external effect occurred.
More partitions can increase parallel consumption and storage distribution. They also increase metadata, open files, replication work, and the number of independent order domains. Partition count is therefore an operating decision, not a magic throughput setting.
Keys define an ordering domain
When all events for one order use orderId as the key and the producer’s partitioning remains compatible, those records can land in the same partition. That supports order for that entity. It does not order different orders against one another, and changing partition count can change the partition chosen by common hash-based strategies.
An empty or unstable key can spread related records across partitions. A hot key can overload one partition even when the topic has spare capacity elsewhere. Select keys from correctness and load evidence, then test their real distribution.
Consumer groups and offsets
Consumers with the same group ID cooperate. Traditional group assignment gives each partition to one member at a time; consumers beyond the partition count may be idle. Different groups read independently and maintain different offsets, which is useful when billing, search indexing, and analytics each need the stream.
Offset commit is the group’s recorded progress. Committing before the durable business effect risks loss after a crash. Committing after the effect risks redelivery. The latter is commonly safer when the handler is idempotent, because repeated work can be detected while missing work cannot be reconstructed from an offset that advanced too early.
Rebalances are ownership changes
Consumers join, leave, stall, or change subscriptions, so the group can revoke and reassign partitions. A consumer must stop using revoked ownership, finish or abort bounded work, and commit only offsets whose associated effects are durable. Long processing inside the polling thread can trigger liveness problems unless poll and processing budgets are designed together.
Static membership or cooperative assignment can reduce disruption in some deployments, but they do not remove failures or make application effects atomic. Treat every rebalance-related feature as a versioned operational mechanism and verify it against the selected client and broker versions.
Spring consumer example
@KafkaListener(topics = "order-events", groupId = "inventory-projection")
public void onOrderEvent(OrderEvent event) {
projectionService.applyOnce(event.eventId(), event.orderId(), event.version());
}The annotation does not define the full guarantee. applyOnce still needs a database uniqueness invariant or equivalent durable deduplication, and the container’s acknowledgment and transaction configuration must match the intended crash behavior.
Failure scenario
A consumer updates the projection and crashes before its offset is committed. The partition is reassigned and the record is delivered again. Without a stable event ID or version guard, the second execution can send a duplicate email or apply an increment twice. With a same-transaction deduplication record, the replay becomes a verified no-op.
Common mistakes
- Claiming topic-wide order when the topic has multiple partitions.
- Scaling consumers above partitions and expecting more traditional group parallelism.
- Using offset commit as proof of external API success.
- Choosing a high-cardinality key without checking hot-entity traffic.
- Performing unbounded work between polls and blaming rebalances on the broker alone.
Production validation
Test key distribution, per-partition throughput, consumer lag, rebalance duration, duplicate delivery, poison records, and crash recovery. Kill a consumer after its database commit but before offset commit. Confirm a new owner reprocesses safely. Add and remove consumers under load, verify no partition is processed concurrently by stale ownership, and correlate business completion separately from offset movement.
Sources
- Apache Kafka 4.1, “Design,” accessed 2026-08-18: https://kafka.apache.org/41/design/design/.
- Apache Kafka documentation, accessed 2026-08-18: https://kafka.apache.org/documentation/.
Related reading
Use the Event-Driven Systems Learning Path for the full sequence. Continue with Reliable Spring Kafka Consumers, Message Delivery Semantics, and Event Ordering and State Convergence. Browse the topic cluster for the complete map.