Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
Row-Oriented vs Columnar Storage
Was this section helpful?
- C-Store: A Column-Oriented DBMS, Michael Stonebraker, Daniel J. Abadi, Adam Batkin, Xuedong Chen, Mitch Cherniack, Miguel Ferreira, Edmond Lau, Amerson Lin, Sam Madden, Elizabeth O'Neil, Pat O'Neil, Alexander Rasin, Nga Tran, and Stan Zdonik, 2005 Proceedings of the 31st International Conference on Very Large Data Bases (VLDB Endowment) DOI: 10.14778/1060942.1060945 - Defines the original principles of column-oriented database management systems.
- Apache Parquet, Apache Software Foundation, 2024 - Official specification and explanation of the columnar storage format, widely used in data lakes.
- Apache ORC, Apache Software Foundation, 2024 (Apache Software Foundation) - Official documentation for the Optimized Row Columnar (ORC) file format, another prominent columnar choice.
- Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017 (O'Reilly Media) - Provides detailed explanations of storage architectures, including row-oriented and columnar approaches, and their performance implications for OLTP vs. OLAP.
- Apache Avro, Apache Software Foundation, 2024 - Official specification for the row-oriented data serialization framework, particularly useful for managing schema evolution in ingestion layers.
© 2025 ApX Machine LearningEngineered with