Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
File Pruning and Skipping
Was this section helpful?
- Apache Iceberg Documentation, The Apache Software Foundation, 2024 (The Apache Software Foundation) - Explains Apache Iceberg table format concepts, including manifests and data skipping mechanisms.
- Apache Parquet Format Specification, The Apache Software Foundation, 2024 - Details the structure of Parquet files, including metadata organization and column statistics.
- Delta Lake Documentation, The Linux Foundation, 2024 - Covers Delta Lake features for performance, such as data skipping and Z-ordering.
- Apache Iceberg: A Table Format for Analytic Datasets, Ryan Blue, Daniel Weeks, Jeremy Klumpp, Russell Spitzer, Fady Essam, Anton Okolnychyi, 2020 Proceedings of the VLDB Endowment, Vol. 13 (VLDB Endowment) DOI: 10.14778/3407086.3407153 - Presents the architecture and features of Apache Iceberg, focusing on its metadata management with manifest lists and files for efficient query planning and data skipping.
© 2025 ApX Machine LearningEngineered with