Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
Handling Schema Evolution
Was this section helpful?
- Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017 (O'Reilly Media) - Chapter 4, 'Data Encoding and Evolution,' provides a discussion on schema evolution, backward and forward compatibility, and different data formats in distributed systems.
- Apache Iceberg: A Table Format for Analytic Datasets, Ryan Blue, Daniel Weeks, Parosh Jasani, Justin S. Johnson, Andrew Gallagher, Jason D. Reid, Brock Noland, Michael Yoder, Steven Hamm, 2020 Proceedings of the VLDB Endowment (PVLDB), Vol. 13 (VLDB Endowment) DOI: 10.14778/3407914.3407920 - This paper introduces Apache Iceberg, detailing its architecture and how it addresses challenges like schema evolution and hidden partitioning in large-scale data lakes.
- Delta Lake: High-Performance ACID Table Storage for Big Data, Michael Armbrust, Ali Ghodsi, Andrew Liu, Xiangrui Meng, Joseph Bradley, Burak Yavuz, Jeffrey H. Reback, Xiang Zhang, Evan Zhou, 2020 Proceedings of the VLDB Endowment (PVLDB), Vol. 13 (VLDB Endowment) DOI: 10.14778/3400735.3400742 - Presents Delta Lake, an open-source storage layer that brings ACID transactions and schema enforcement to data lakes, aiding schema evolution.
- Structured Streaming Programming Guide, The Apache Software Foundation, 2024 - The sections on managing schema evolution and handling incompatible changes in streaming data pipelines provide practical implementation details for Apache Spark.
© 2025 ApX Machine LearningEngineered with