Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
Object Storage Semantics
Was this section helpful?
- How Amazon S3 Works, Amazon Web Services (AWS), Ongoing (AWS) - Explains the fundamental principles of Amazon S3, including its object model, flat namespace, immutability, consistency guarantees, and operational characteristics.
- Blob storage concepts, Microsoft Azure Documentation, 2023 (Microsoft) - Details the core concepts of Azure Blob Storage, covering containers, blobs, storage structure, and the handling of data operations, complementing S3 documentation.
- Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017 (O'Reilly Media) - Offers a comprehensive look at distributed systems foundations, including storage engine design, consistency models, and the trade-offs essential for understanding object storage.
© 2025 ApX Machine LearningEngineered with