Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
Change Data Capture (CDC)
Was this section helpful?
- Debezium Documentation, The Debezium Community, N/A - Official documentation for Debezium, an open-source platform for change data capture. It covers the setup, configuration, and operational aspects of log-based CDC connectors, including how they handle initial snapshots and continuous event streaming from various databases.
- Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017 (O'Reilly Media) - This book provides an in-depth exploration of the principles behind reliable, scalable, and maintainable data systems. Chapters on replication, transaction logs, and stream processing provide foundational understanding for log-based change data capture mechanisms.
- Delta Lake Documentation, The Delta Lake Community, N/A - Official documentation for Delta Lake, an open-source storage layer that brings ACID transactions to Apache Spark and big data workloads. Relevant for understanding how to perform MERGE INTO operations on data lake tables to apply CDC events.
- Apache Iceberg Documentation, The Apache Iceberg Community, N/A (Apache Software Foundation) - Official documentation for Apache Iceberg, an open table format for huge analytic datasets. It details how Iceberg supports schema evolution, hidden partitioning, and table modifications (including MERGE INTO), which are important for managing CDC-driven updates in a data lake.
© 2025 ApX Machine LearningEngineered with