Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
Caching Strategies
Was this section helpful?
- Computer Architecture: A Quantitative Approach, John L. Hennessy, David A. Patterson, 2019 (Morgan Kaufmann) - This book provides comprehensive coverage of caching hierarchies, performance metrics like effective access time, and common eviction policies such as LRU and LFU.
- Trino Documentation: Hive connector caching, The Trino Community, 2024 (The Trino Community) - Official documentation describing the configuration and operation of I/O caching for the Hive connector in Trino, including details on integration with Rubix.
- Apache Spark Documentation: RDD Persistence, The Apache Spark Project, 2024 - Official documentation explaining how to explicitly cache or persist RDDs and DataFrames in Spark, a fundamental technique for optimizing data access.
- Apache Iceberg Specification, The Apache Iceberg Project, 2024 - Provides the technical specification for Apache Iceberg, detailing its architecture for snapshot isolation and immutable files, which supports cache consistency.
© 2025 ApX Machine LearningEngineered with