Building Scalable Data Warehouses
Chapter 1: MPP Architecture and Storage Internals
Massively Parallel Processing Fundamentals
Decoupling Compute and Storage
Columnar Storage Formats and Compression
Micro-partitioning and Metadata Management
Hands-on practice: Inspecting Storage Profiles
Chapter 2: Advanced Data Modeling at Scale
Dimensional Modeling Constraints in Big Data
Data Vault 2.0 Implementation Patterns
Handling Semi-Structured Data
Schema Evolution and Versioning
Hands-on practice: Designing a Data Vault
Chapter 3: High-Throughput Ingestion Pipelines
Change Data Capture Architectures
Idempotency in Data Pipelines
Micro-batch vs Streaming Ingestion
Handling Late Arriving Data
Hands-on practice: Building a CDC Pipeline
Chapter 4: Query Optimization and Performance Tuning
Analyzing Query Execution Plans
Partition Pruning and Clustering Keys
Join Strategies: Broadcast vs Shuffle
Materialized Views and Caching Layers
Hands-on practice: Tuning High-Latency Queries
Chapter 5: Governance, Security, and Observability
Role-Based Access Control Hierarchies
Dynamic Data Masking and Tokenization
Row-Level Security Implementation
Resource Monitoring and Cost Control
Hands-on practice: Configuring Security Policies
Handling Late Arriving Data
Was this section helpful?
- The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Unified Stream Processing, Tyler Akidau, Alex Balikov, Kaya Bekiroğlu, Slava Chernyak, Josh Haberman, Reuven Lax, Sam McVeety, Daniel Mills, Fraser Nevard, Oleksandr Shvets, Sam Whittle, 2017 Proceedings of the VLDB Endowment, Vol. 10 (VLDB Endowment) DOI: 10.14778/3137628.3137637 - This paper introduces the Dataflow Model, which provides a unified approach to batch and stream processing, defining event time, processing time, and watermarks for managing temporal challenges.
- Big Data: Principles and Best Practices of Scalable Realtime Data Systems, Nathan Marz and James Warren, 2015 (Manning Publications) - Presents the Lambda Architecture, a dual-layer approach for handling data, including late-arriving events through its batch layer, contrasting it with Kappa.
- Bitemporal Data: Theory and Practice, Tom Johnston, 2014 (Morgan Kaufmann) - A comprehensive book on bitemporal data modeling, explaining how to store and query data that changes over time, including both event time and recording time.
© 2025 ApX Machine LearningEngineered with