Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
The Medallion Architecture
Was this section helpful?
- Medallion Architecture | Databricks Documentation, Databricks, 2024 (Databricks) - Official documentation detailing the Medallion Architecture, its three layers (Bronze, Silver, Gold), and practices for implementing it within a data lakehouse.
- Delta Lake Documentation, The Delta Lake Project, 2025 (The Linux Foundation Projects) - Provides official information on Delta Lake, an open-source storage layer that brings ACID transactions and reliability to data lakes, often used for implementing the Silver layer in a Medallion Architecture.
- Fundamentals of Data Engineering: Planning and Building Robust Data Systems, Joe Reis, Matt Housley, 2022 (O'Reilly Media) - A comprehensive book covering modern data engineering principles, patterns, and architectural considerations, including data lake design and data quality strategies relevant to the Medallion Architecture.
© 2025 ApX Machine LearningEngineered with