Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
Hands-on Practical: Query Analysis
Was this section helpful?
- Apache Parquet, The Apache Software Foundation, 2024 - Explains the columnar storage format, its benefits, and how it enables efficient reads for analytical queries through features like column projection and vectorization.
- Spark SQL Performance Tuning, The Apache Software Foundation, 2024 - Official guide for optimizing Spark SQL queries, including generating and interpreting execution plans, using partitioning for improved performance, and managing shuffle operations.
- Building a Data Lakehouse with Delta Lake, Vini J. Varghese, Andy Feng, Deniz Ilkbasaran, 2023 (O'Reilly Media) - Discusses modern data lake architectures and optimization strategies for large-scale data, including effective partitioning and file management techniques that enhance query performance.
© 2025 ApX Machine LearningEngineered with