Building Scalable Data Warehouses
Chapter 1: MPP Architecture and Storage Internals
Massively Parallel Processing Fundamentals
Decoupling Compute and Storage
Columnar Storage Formats and Compression
Micro-partitioning and Metadata Management
Hands-on practice: Inspecting Storage Profiles
Chapter 2: Advanced Data Modeling at Scale
Dimensional Modeling Constraints in Big Data
Data Vault 2.0 Implementation Patterns
Handling Semi-Structured Data
Schema Evolution and Versioning
Hands-on practice: Designing a Data Vault
Chapter 3: High-Throughput Ingestion Pipelines
Change Data Capture Architectures
Idempotency in Data Pipelines
Micro-batch vs Streaming Ingestion
Handling Late Arriving Data
Hands-on practice: Building a CDC Pipeline
Chapter 4: Query Optimization and Performance Tuning
Analyzing Query Execution Plans
Partition Pruning and Clustering Keys
Join Strategies: Broadcast vs Shuffle
Materialized Views and Caching Layers
Hands-on practice: Tuning High-Latency Queries
Chapter 5: Governance, Security, and Observability
Role-Based Access Control Hierarchies
Dynamic Data Masking and Tokenization
Row-Level Security Implementation
Resource Monitoring and Cost Control
Hands-on practice: Configuring Security Policies
Micro-partitioning and Metadata Management
Was this section helpful?
- The Snowflake Elastic Data Warehouse, Benoit Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, Allison W. Lee, Ashish Motivala, Abdul Q. Munir, Steven Pelley, Peter Povinec, Greg Rahn, Spyridon Triantafyllis, Philipp Unterbrunner, 2016 Proceedings of the 2016 International Conference on Management of Data (ACM) DOI: 10.1145/2882903.2903746 - Academic paper detailing the core architecture of Snowflake, including micro-partitions, metadata service, and distributed query execution.
- Understanding Snowflake's Unique Architecture, Snowflake Documentation, 2024 (Snowflake Inc.) - Official guide to Snowflake's cloud-native architecture, explaining micro-partitions, data storage, and query optimization.
- BigQuery architecture overview, Google Cloud Documentation, 2024 (Google LLC) - Overview of BigQuery's storage model, describing its approach to columnar data, automatic partitioning, and metadata-based pruning.
- Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017 (O'Reilly Media) - Discusses fundamental concepts of data storage strategies, immutable data structures, and distributed systems relevant to modern data warehouse design.
© 2025 ApX Machine LearningEngineered with