ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Real-Time Data Pipelines with Apache Kafka and Flink
Chapter 1: Stream Processing Architectures and Semantics
Evolution of Distributed Logs
Lambda versus Kappa Architecture
Processing Guarantees and Semantics
Event Time versus Processing Time
Hands-on Practical: Designing a Kappa Pipeline
Chapter 2: Advanced Kafka Producer and Consumer Internals
Replication Protocols and ISRs
Idempotent Producers and Transactions
Custom Partitioning Strategies
Consumer Group Rebalancing protocols
Hands-on Practical: Implementing Transactional Writes
Chapter 3: Flink State Management and Checkpointing
State Backends: HashMap versus RocksDB
Asynchronous Barrier Snapshots
Incremental Checkpointing
State Schema Evolution
Hands-on Practical: State Migration with Savepoints
Chapter 4: Advanced Windowing and Time Attributes
Watermark Generation Strategies
Handling Late Data and Side Outputs
Custom Window Triggers and Evictors
Session Windows and Gap Analysis
Hands-on Practical: Building a Custom Trigger
Chapter 5: Low-Level Operations with ProcessFunctions
The ProcessFunction Hierarchy
Timer Services and Event Scheduling
Broadcast State Pattern
Async I/O for External Lookups
Hands-on Practical: Dynamic Rule Evaluation
Chapter 6: Production Deployment and Reliability
Serialization with Avro and Protobuf
Schema Registry Integration
Kafka Connect for Sink and Source
Failure Recovery Strategies
Hands-on Practical: Schema Evolution in Flight
Chapter 7: Performance Tuning and Monitoring
Identifying Backpressure
Memory Management and Slot Allocation
Tuning RocksDB Performance
Kafka Consumer Lag Analysis
Hands-on Practical: Resolving Skewed Data
Chapter 8: Real-Time AI and Feature Engineering
Online Feature Generation
Model Serving Patterns in Streams
Request-Response over Async Streams
Feature Store Integration
Hands-on Practical: Real-Time Inference Pipeline

Watermark Generation Strategies

Watermarks serve as the pacing mechanism for event time processing. In a distributed streaming environment, they solve the problem of out-of-order data by providing a metric of progress. A watermark carrying a timestamp ttt declares that the stream is complete up to time ttt. This assertion allows window operators to close buckets and trigger calculations, assuming that no further events with a timestamp t′≤tt' \le tt′≤t will arrive.

This mechanism is heuristic by definition. Because network partitions and variable latency make perfect knowledge of the stream impossible, a watermark strategy represents a tradeoff between latency and correctness. An aggressive strategy that generates watermarks close to the current processing time yields low latency results but risks dropping valid data as "late". A conservative strategy waits longer for straggling events, ensuring higher accuracy at the cost of increased latency.

The Watermark Strategy Interface

In Flink, the WatermarkStrategy defines how watermarks are generated and how timestamps are assigned to events. This interface decouples the logic of timestamp extraction from the logic of watermark emission. While simple applications might rely on predefined strategies, production pipelines often require tuning these parameters to match the specific skew characteristics of the upstream data source, such as Kafka partitions.

There are two primary styles of watermark generation: periodic and punctuated.

Periodic Watermarks

Periodic generation is the standard approach for high-throughput streams. The system emits a watermark at regular processing time intervals, defined by the auto-watermark interval. This interval is decoupled from the event arrival rate. Even if thousands of events arrive per second, the watermark might only update every 200 milliseconds to reduce overhead.

The framework calls the onPeriodicEmit method of the WatermarkGenerator at these intervals. The generator inspects the maximum timestamp seen so far and emits a watermark based on a lag tolerance.

Punctuated Watermarks

Punctuated generation relies on the data itself. The generator observes every incoming event and may choose to emit a watermark immediately. This is necessary in scenarios with sparse data or when specific marker records indicate the completion of a logical block, such as an "end-of-batch" signal in a hybrid stream. This approach incurs higher overhead because the onEvent method must evaluate the watermark logic for every record.

Bounded Out-of-Orderness

The most frequent pattern in distributed logs like Kafka is Bounded Out-of-Orderness. This strategy assumes that while elements move out of sequence, they do so within a finite time window. If the maximum delay in the system is known to be 5 seconds, the watermark at any point is the maximum timestamp observed minus 5 seconds.

W=max⁡(Tevent)−TtoleranceW = \max(T_{event}) - T_{tolerance}W=max(Tevent​)−Ttolerance​

Events arriving within this tolerance buffer are processed correctly. Events arriving after the watermark passes the event's timestamp are late.

The following chart illustrates the relationship between event time, processing time, and the watermark line. Points below the watermark line represent late events that would be dropped or sent to a side output.

The blue line represents the watermark horizon. Green points are processed within the window. Red points arrive after the watermark has passed their timestamp, classifying them as late.

Implementing a Custom Strategy

While Flink provides WatermarkStrategy.forBoundedOutOfOrderness, complex AI pipelines often require custom logic. For example, you might need a strategy that advances time based on the average of the last NNN timestamps rather than the strict maximum to filter out outliers, or a strategy that handles distinct logical streams interleaved in a single partition.

To implement a custom strategy, you define a WatermarkGenerator.

public class SmoothLagGenerator implements WatermarkGenerator<MyEvent> {
    
    private final long maxLag;
    private long currentMaxTimestamp;

    public SmoothLagGenerator(long maxLag) {
        this.maxLag = maxLag;
        this.currentMaxTimestamp = Long.MIN_VALUE + maxLag + 1;
    }

    @Override
    public void onEvent(MyEvent event, long eventTimestamp, WatermarkOutput output) {
        // Track the maximum timestamp seen
        currentMaxTimestamp = Math.max(currentMaxTimestamp, eventTimestamp);
    }

    @Override
    public void onPeriodicEmit(WatermarkOutput output) {
        // Emit the watermark minus the lag
        output.emitWatermark(new Watermark(currentMaxTimestamp - maxLag));
    }
}

This generator is then attached to the stream source:

WatermarkStrategy<MyEvent> strategy = WatermarkStrategy
    .forGenerator(ctx -> new SmoothLagGenerator(Duration.ofSeconds(5).toMillis()))
    .withTimestampAssigner((event, timestamp) -> event.getCreationTime());

DataStream<MyEvent> stream = env.fromSource(kafkaSource, strategy, "Kafka Source");

Watermark Propagation and Synchronization

In a parallel streaming topology, watermarks must be synchronized across partitions. A stream operator, such as a Window ProcessFunction, usually receives input from multiple parallel instances of an upstream operator. Each upstream instance emits its own watermark based on the data it observes.

The downstream operator maintains a partition-specific watermark for each input channel. The operator's logical time is defined as the minimum of all input channel watermarks. This ensures that the operator does not advance its time past the "slowest" upstream partition.

Woperator=min⁡(Wpartition_0,Wpartition_1,…,Wpartition_n)W_{operator} = \min(W_{partition\_0}, W_{partition\_1}, \dots, W_{partition\_n})Woperator​=min(Wpartition_0​,Wpartition_1​,…,Wpartition_n​)

This mechanism creates a dependency on the slowest partition. If one Kafka partition lags significantly due to network issues or data skew, the entire windowing operation for that operator stalls.

The window operator determines its global watermark by taking the minimum of all incoming channel watermarks. Here, Partition 2 holds back the global progress to 12:02.

Handling Idle Sources

A critical edge case in watermark propagation is the idle source problem. If a specific Kafka partition contains no data for an extended period, it will not emit watermarks. Because the downstream operator calculates the minimum across all partitions, the absence of a watermark from the idle partition prevents the minimum from increasing. The entire application's time effectively freezes, and windows do not close.

This often occurs in topic partitions reserved for low-frequency categories or during periods of low traffic. To resolve this, Flink provides an idleness detection mechanism.

WatermarkStrategy<MyEvent> strategy = WatermarkStrategy
    .forBoundedOutOfOrderness(Duration.ofSeconds(5))
    .withIdleness(Duration.ofMinutes(1));

When withIdleness is configured, the system marks a partition as "idle" if no events arrive within the specified duration. Once marked idle, the partition is excluded from the downstream watermark calculation. This allows the global watermark to advance based solely on the active partitions. When data resumes flowing from the idle partition, it resumes active status and once again contributes to the synchronization logic.

Careful configuration of the idleness timeout is necessary. If set too short, the system might flip-flop between idle and active states frequently. If set too long, the pipeline latency increases significantly when a partition actually goes silent. This configuration is a mandatory check for production readiness in any environment where data rates vary across partitions.

  • Working with Event Time, The Apache Flink Community, 2018 (Apache Software Foundation) - Official documentation for Apache Flink's event time processing, watermarks, watermark strategies, and idleness handling.
  • Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing, Tyler Akidau, Slava Chernyak, Reuven Lax, 2018 (O'Reilly Media) - A comprehensive guide covering the fundamental concepts of distributed stream processing, including event time, watermarks, windowing, and correctness guarantees.
  • The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Unified Stream Processing, Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J. Fernández-Moctezuma, Rui Li, Davorin Marković, Francis McCabe, Jean-Baptiste Onofré, Abigail P. O'Neil, Paul Nordstrom, and Sam Whittle, 2015 Proceedings of the VLDB Endowment, Vol. 8 (VLDB Endowment) DOI: 10.14778/2824032.2824076 - Foundational paper introducing the Dataflow Model, which influenced modern stream processing systems like Flink, detailing event time, processing time, and watermark semantics.

© 2026 ApX Machine Learning