Distributed systems operate on the assumption that hardware eventually fails. A disk failure in a single-node database often results in data loss. To mitigate this, Apache Kafka relies on partition replication to guarantee data persistence. However, simply copying data to multiple nodes is insufficient for high-throughput environments. Understanding the specific mechanics of how data propagates from a leader to its followers and how the cluster determines when a message is safely "committed" is essential.
Every partition in Kafka has a single replica designated as the Leader. All produce and consume requests (by default) go through this leader. The remaining replicas are Followers.
Followers in Kafka are passive in terms of cluster orchestration but active in terms of data retrieval. They operate essentially as specialized consumers. A follower sends FetchRequest messages to the leader, requesting data starting from a specific offset. The leader returns the data, and the follower appends it to its local log. This "pull" model prevents the leader from overwhelming followers if they cannot keep up with the ingestion rate.
The interaction is critical for performance. The leader does not push data; it waits for the follower to ask for it. This mechanism allows Kafka to handle followers with different network capabilities or disk I/O speeds without stalling the leader, provided the durability settings do not strictly enforce synchronous replication.
Not all replicas are equal. Kafka maintains a dynamic set of replicas called the In-Sync Replicas (ISR). This list includes the leader and any followers that are functionally "caught up" with the leader.
Historically, Kafka defined "caught up" using both message count lag and time lag. Modern versions rely almost exclusively on time. A follower is considered in-sync if it has sent a FetchRequest to the leader within the window defined by replica.lag.time.max.ms.
If a follower fails to make a fetch request or if it makes requests but cannot catch up to the leader's Log End Offset (LEO) within this time window, the leader evicts the follower from the ISR. The follower remains a replica, but it is effectively "out of sync."
To understand how Kafka guarantees consistency, we must distinguish between two pointers maintained for every partition replica:
The High Watermark acts as the barrier for consumers. A consumer can only read messages up to the High Watermark. Messages between the High Watermark and the Leader's LEO are considered "uncommitted" and are not exposed to consumers. This prevents the "phantom read" problem where a consumer reads a message that is later lost due to a leader failure.
The High Watermark is calculated by the leader. It effectively tracks the minimum LEO among the ISR set.
When a producer sends a message, it is appended to the leader's log (incrementing the leader's LEO). The followers fetch this data and update their own LEOs. Once the leader receives fetch requests indicating that all ISR members have the message, the leader advances the High Watermark.
The leader tracks the replication status of all followers. In this state, the High Watermark is 100 because Follower 2 has not yet acknowledged offset 101. Follower 3 is likely outside the ISR if its lag exceeds the time threshold.
The strength of the durability guarantee depends on the producer's acks configuration and the broker's min.insync.replicas setting.
When a producer sets acks=all (or -1), the leader waits until the message is written to the local log of all current members of the ISR before sending an acknowledgment. This provides the strongest durability guarantee. If the leader crashes, any specific follower in the ISR is eligible to become the new leader without data loss.
However, acks=all alone does not guarantee data safety if the ISR shrinks to just the leader. If all followers fall behind or crash, the ISR set size becomes 1. The leader acknowledges the write immediately after writing to its own disk. If that single leader fails, the data is lost.
To prevent this, you must configure min.insync.replicas at the broker or topic level. This setting acts as a gatekeeper.
If the number of available in-sync replicas falls below this threshold, the leader rejects produce requests with acks=all, throwing a NotEnoughReplicasException. This effectively stops the pipeline to preserve data consistency.
This configuration introduces a classic distributed systems trade-off between Availability and Consistency.
min.insync.replicas = 1. The cluster accepts writes as long as at least one node (the leader) is alive. Risk: Data loss if the leader fails before replication.min.insync.replicas = 2 (in a replication factor 3 setup). The cluster requires at least two nodes to acknowledge the write. Risk: If two nodes fail (or undergo maintenance), the partition becomes read-only (unavailable for writes).The following chart visualizes the relationship between replication latency and producer throughput relative to the number of required acknowledgments.
As throughput increases, the latency cost of requiring synchronicity across all replicas (
acks=all) grows significantly compared to leader-only acknowledgment.
A catastrophic scenario occurs when all replicas in the ISR fail. The cluster is left with followers that were outside the ISR (lagging) or no online replicas at all.
If a lagging follower eventually comes back online, Kafka faces a dilemma managed by the unclean.leader.election.enable configuration:
For financial or audit-trail pipelines where data integrity is absolute, this setting must remain false. For metrics or logging streams where uptime is more critical than a few missing logs, true might be acceptable.
Replication traffic competes with producer and consumer traffic for network bandwidth. In high-performance tuning, you might isolate replication traffic to a separate network interface if your hardware allows it, though this requires advanced broker configuration.
More commonly, you will tune the fetch size. The replica.fetch.max.bytes setting controls how much data a follower attempts to fetch in a single request. Increasing this improves throughput for high-volume topics but increases memory pressure on the broker and can cause latency spikes if the network is the bottleneck.
Understanding these internal protocols allows you to architect systems that withstand node failures without manual intervention while predicting exactly how much latency is introduced by your consistency requirements.
Was this section helpful?
© 2026 ApX Machine LearningEngineered with