Building Scalable Data Warehouses
Chapter 1: MPP Architecture and Storage Internals
Massively Parallel Processing Fundamentals
Decoupling Compute and Storage
Columnar Storage Formats and Compression
Micro-partitioning and Metadata Management
Hands-on practice: Inspecting Storage Profiles
Chapter 2: Advanced Data Modeling at Scale
Dimensional Modeling Constraints in Big Data
Data Vault 2.0 Implementation Patterns
Handling Semi-Structured Data
Schema Evolution and Versioning
Hands-on practice: Designing a Data Vault
Chapter 3: High-Throughput Ingestion Pipelines
Change Data Capture Architectures
Idempotency in Data Pipelines
Micro-batch vs Streaming Ingestion
Handling Late Arriving Data
Hands-on practice: Building a CDC Pipeline
Chapter 4: Query Optimization and Performance Tuning
Analyzing Query Execution Plans
Partition Pruning and Clustering Keys
Join Strategies: Broadcast vs Shuffle
Materialized Views and Caching Layers
Hands-on practice: Tuning High-Latency Queries
Chapter 5: Governance, Security, and Observability
Role-Based Access Control Hierarchies
Dynamic Data Masking and Tokenization
Row-Level Security Implementation
Resource Monitoring and Cost Control
Hands-on practice: Configuring Security Policies
Micro-batch vs Streaming Ingestion
Was this section helpful?
- Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017 (O'Reilly Media) - Explores core concepts of batch and stream processing, distributed systems, consistency models, and data storage, providing a comprehensive background for understanding ingestion challenges.
- C-Store: A Column-Oriented DBMS, Michael Stonebraker, Daniel J. Abadi, Adam Batkin, Xuedong Chen, Mitch Cherniack, Miguel Ferreira, Edmond Lau, Amerson Lin, Samuel Madden, Elizabeth J. O'Neil, Patrick E. O'Neil, Alexander Rasin, Nga Tran, Stanley B. Zdonik, 2005 Proceedings of the 31st International Conference on Very Large Data Bases (VLDB) (ACM) DOI: 10.14778/1083592.1083658 - Introduces the design principles of column-oriented database management systems, which are foundational to modern analytical data warehouses and explain their optimization for read-heavy workloads and the implications for writing.
- Snowpipe Streaming API, Snowflake, 2024 (Snowflake) - Provides practical details on how a leading cloud data warehouse implements low-latency, high-throughput streaming ingestion into columnar storage, addressing the row-oriented write-ahead log (WAL) and background merger described.
- The Log: What every software engineer should know about real-time data's unifying abstraction, Jay Kreps, 2013 (LinkedIn Engineering Blog) - Explains the foundational role of transaction logs and change data capture (CDC) in modern data systems, linking directly to the concept of 'reliable change data capture (CDC)' in the section.
© 2025 ApX Machine LearningEngineered with