Real-Time Data Pipelines with Apache Kafka and Flink
Chapter 1: Stream Processing Architectures and Semantics
Evolution of Distributed Logs
Lambda versus Kappa Architecture
Processing Guarantees and Semantics
Event Time versus Processing Time
Hands-on Practical: Designing a Kappa Pipeline
Chapter 2: Advanced Kafka Producer and Consumer Internals
Replication Protocols and ISRs
Idempotent Producers and Transactions
Custom Partitioning Strategies
Consumer Group Rebalancing protocols
Hands-on Practical: Implementing Transactional Writes
Chapter 3: Flink State Management and Checkpointing
State Backends: HashMap versus RocksDB
Asynchronous Barrier Snapshots
Incremental Checkpointing
State Schema Evolution
Hands-on Practical: State Migration with Savepoints
Chapter 4: Advanced Windowing and Time Attributes
Watermark Generation Strategies
Handling Late Data and Side Outputs
Custom Window Triggers and Evictors
Session Windows and Gap Analysis
Hands-on Practical: Building a Custom Trigger
Chapter 5: Low-Level Operations with ProcessFunctions
The ProcessFunction Hierarchy
Timer Services and Event Scheduling
Broadcast State Pattern
Async I/O for External Lookups
Hands-on Practical: Dynamic Rule Evaluation
Chapter 6: Production Deployment and Reliability
Serialization with Avro and Protobuf
Schema Registry Integration
Kafka Connect for Sink and Source
Failure Recovery Strategies
Hands-on Practical: Schema Evolution in Flight
Chapter 7: Performance Tuning and Monitoring
Identifying Backpressure
Memory Management and Slot Allocation
Tuning RocksDB Performance
Kafka Consumer Lag Analysis
Hands-on Practical: Resolving Skewed Data
Chapter 8: Real-Time AI and Feature Engineering
Online Feature Generation
Model Serving Patterns in Streams
Request-Response over Async Streams
Feature Store Integration
Hands-on Practical: Real-Time Inference Pipeline
Feature Store Integration
Was this section helpful?
- Michelangelo: Uber's Machine Learning Platform, Siby, Praveen and Goel, Anirudh and Jain, Manu and Jain, Amit and Singh, Amit and Sharma, Hien and Kumar, Karan and Panwar, Vaibhav and Srivatsa, Sathish and Shvachko, Alex and Grewal, Navjot and Arora, Sahitya and Kumar, Manish and Singh, Kamesh and Singh, Gaurav and Pudi, Venkata and Nallamothu, Venkata and Jain, Anupam and Gupta, Sanjai, 2017 arXiv preprint arXiv:1710.05370 - Describes the architecture of a large-scale machine learning platform, including the necessity and design of a feature store to address training-serving skew and ensure feature consistency.
- Asynchronous I/O for External Data Access, Apache Flink Documentation, 2024 - Official documentation explaining Flink's Asynchronous I/O API, which is crucial for high-throughput interaction with external online feature stores.
- Designing Machine Learning Systems, Huyen, Chip, 2022 (O'Reilly Media) - Provides a comprehensive explanation of feature stores, covering their architecture, role in preventing training-serving skew, and requirements like point-in-time correctness for ML systems.
© 2025 ApX Machine LearningEngineered with