Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
Compression Algorithms
Was this section helpful?
- Apache Parquet Format, Apache Software Foundation, 2025 (Apache Software Foundation) - Describes the official specification for the Apache Parquet columnar file format, detailing its structure and the integration of compression codecs for splittability and parallel processing.
- Zstandard: Fast Real-time Compression Algorithm, Yann Collet, Pascal Massimino, Jean-Marc Odobez, 2016 USENIX Annual Technical Conference (ATC) (USENIX Association) DOI: 10.5555/3008035.3008044 - Presents the Zstandard compression algorithm, detailing its design for achieving high compression ratios with fast decompression speeds, a significant improvement over prior codecs.
- Snappy: A Fast Compressor/Decompressor, Google, 2024 - The official project repository for Google Snappy, describing its design principles for high compression and decompression speed with practical compression ratios, suitable for interactive data processing.
- Designing Data-Intensive Applications, Martin Kleppmann, 2017 (O'Reilly Media) - A comprehensive book on the fundamental principles of designing scalable data systems, covering data storage, processing, and distributed architectures pertinent to data lake optimization.
© 2025 ApX Machine LearningEngineered with