Introduction to Data Lake Architectures
Chapter 1: Architectural Foundations
Decoupling Compute and Storage
Object Storage Semantics
The Medallion Architecture
Lambda and Kappa Architectures
Chapter 2: File Formats and Optimization
Row-Oriented vs Columnar Storage
Apache Parquet Internals
Open Table Formats
Compression Algorithms
Partitioning Strategies
Hands-on Practical: Optimizing File Layouts
Chapter 3: Ingestion Pipelines
Batch Ingestion Workflows
Change Data Capture (CDC)
Handling Schema Evolution
The Small File Problem
Idempotency in Pipelines
Hands-on Practical: Building a CDC Pipeline
Chapter 4: Metadata and Cataloging
The Role of the Metastore
Partition Discovery
Technical Governance
Data Lineage Implementation
Hands-on Practical: Configuring a Catalog
Chapter 5: Querying and Performance
Distributed Query Engines
File Pruning and Skipping
Vectorized Query Execution
Caching Strategies
Hands-on Practical: Query Analysis
Technical Governance
Was this section helpful?
- AWS Lake Formation Developer Guide, Amazon Web Services, 2024 (Amazon Web Services) - An official guide demonstrating data lake governance implementation, including dual-layer security, Role-Based Access Control, fine-grained access control (Row-Level Security/Column-Level Security), and the token vending pattern.
- Unity Catalog documentation, Databricks, 2025 (Databricks) - Official documentation for unified data governance on the Databricks Lakehouse Platform, outlining features such as RBAC, Row-Level Security, Column-Level Security, and tag-based policies.
- Guide to Attribute Based Access Control (ABAC) Definition and Considerations, Vincent Hu, David Ferraiolo, David Kuhn, Adam Schnitzer, 2013 NIST Special Publication 800-162 (National Institute of Standards and Technology (NIST)) DOI: 10.6028/NIST.SP.800-162 - A foundational guide from NIST defining Attribute-Based Access Control, which provides the underlying principles for tag-based governance discussed in modern data lake architectures.
© 2025 ApX Machine LearningEngineered with