Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
Vectorized Query Execution
Was this section helpful?
- MonetDB/X100: Hyper-Pipelining Query Execution, Peter A. Boncz, Marcin Zukowski, Niels Nes, 2005 Proceedings of CIDR 2005, Second Biennial Conference on Innovative Data Systems Research (ACM) DOI: 10.1145/1066164.1066244 - This paper introduces hyper-pipelining and vectorized query execution in a column-store context, elaborating on its advantages for CPU utilization and cache efficiency.
- Apache Arrow: A Cross-Language Development Platform for In-Memory Data, Jacques Nadeau, Wes McKinney, Brian Hulette, Julian Hyde, Philipp Hoffmann, Andy Prowell, Uwe L. Korn, Siddharth Teotia, Greg Rahn, Mike Becker, 2018 Proceedings of the 2018 International Conference on Management of Data (SIGMOD) (ACM) DOI: 10.1145/3183713.3196924 - Describes the design and value of Apache Arrow, a key technology for standardized in-memory columnar data, which facilitates efficient vectorized processing and data exchange across systems.
- The Internals of Modern Data Analytics Engines, Matteo Hessel, Julian Kunkel, Tobias M. J. W. van der Heijden, 2023 ACM Computing Surveys (ACM) DOI: 10.1145/3602509 - Provides a current overview of techniques found in modern data analytics engines, including vectorized execution, columnar storage, and distributed processing, providing a broader context for the section's content.
© 2025 ApX Machine LearningEngineered with