Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
Data Lineage Implementation
Was this section helpful?
- OpenLineage Specification, OpenLineage Community, 2024 - A reference defining the open standard for capturing, exchanging, and storing data lineage metadata.
- Monitoring Applications - Spark Listeners, Apache Software Foundation, 2023 - Covers the SparkListener interface, which allows for custom instrumentation and metadata capture during job execution.
- Fundamentals of Data Engineering: Planning and Building Data Systems, Joe Reis, Matt Housley, 2022 (O'Reilly Media) - Offers a broad perspective on data engineering principles, including metadata management and data lineage in contemporary data architectures.
- How dbt works, dbt Labs, 2024 (dbt Labs, Inc.) - Explains how dbt constructs a directed acyclic graph (DAG) of data models and transformations, which inherently facilitates static lineage.
© 2025 ApX Machine LearningEngineered with