Real-Time Data Pipelines with Apache Kafka and Flink
Chapter 1: Stream Processing Architectures and Semantics
Evolution of Distributed Logs
Lambda versus Kappa Architecture
Processing Guarantees and Semantics
Event Time versus Processing Time
Hands-on Practical: Designing a Kappa Pipeline
Chapter 2: Advanced Kafka Producer and Consumer Internals
Replication Protocols and ISRs
Idempotent Producers and Transactions
Custom Partitioning Strategies
Consumer Group Rebalancing protocols
Hands-on Practical: Implementing Transactional Writes
Chapter 3: Flink State Management and Checkpointing
State Backends: HashMap versus RocksDB
Asynchronous Barrier Snapshots
Incremental Checkpointing
State Schema Evolution
Hands-on Practical: State Migration with Savepoints
Chapter 4: Advanced Windowing and Time Attributes
Watermark Generation Strategies
Handling Late Data and Side Outputs
Custom Window Triggers and Evictors
Session Windows and Gap Analysis
Hands-on Practical: Building a Custom Trigger
Chapter 5: Low-Level Operations with ProcessFunctions
The ProcessFunction Hierarchy
Timer Services and Event Scheduling
Broadcast State Pattern
Async I/O for External Lookups
Hands-on Practical: Dynamic Rule Evaluation
Chapter 6: Production Deployment and Reliability
Serialization with Avro and Protobuf
Schema Registry Integration
Kafka Connect for Sink and Source
Failure Recovery Strategies
Hands-on Practical: Schema Evolution in Flight
Chapter 7: Performance Tuning and Monitoring
Identifying Backpressure
Memory Management and Slot Allocation
Tuning RocksDB Performance
Kafka Consumer Lag Analysis
Hands-on Practical: Resolving Skewed Data
Chapter 8: Real-Time AI and Feature Engineering
Online Feature Generation
Model Serving Patterns in Streams
Request-Response over Async Streams
Feature Store Integration
Hands-on Practical: Real-Time Inference Pipeline
Timer Services and Event Scheduling
Was this section helpful?
- Streaming Systems: The What, Where, When, and How of Real-Time Data Processing, Tyler Akidau, Slava Chernyak, Reuven Lax, 2018 (O'Reilly Media) - Explains the foundational concepts of stream processing, including event time, processing time, and watermarks, which are essential for understanding time-dependent logic in Flink.
- Apache Flink: Stream and Batch Processing in a Single Engine, Fabian Hueske, Aljoscha Krettek, Robert Metzger, Till Rohrmann, Stefan Richter, Arne Simon, Kostas Tzoumas, and Stephan Ewen, 2015 Proceedings of the 7th Biennial Conference on Innovative Data Systems Research (CIDR '15) DOI: 10.5441/002/cidr15-05 - Introduces the Flink architecture and its core capabilities for stateful stream processing and time management.
- Stream Processing with Apache Flink: Fundamentals, Streamlining Dataflow, and Advanced Applications, Fabian Hueske, Vasia Kalavri, 2019 (O'Reilly Media) - Provides detailed explanations of Flink's ProcessFunction and TimerService, including practical advice on state and time management.
© 2025 ApX Machine LearningEngineered with