By this stage in the architecture, you have configured object storage and established ingestion pipelines. Your data resides in buckets, likely formatted as Parquet or Avro. However, a distributed query engine cannot execute SQL against these raw files without specific instructions. It requires a schema definition and a directory of file locations to treat loose objects as structured tables.

This chapter defines the metadata and cataloging layer, which acts as the interface between physical storage and compute engines. You will study how the Hive Metastore and AWS Glue Data Catalog maintain the state of a data lake. We will demonstrate how these systems map a logical table, such as sales_data, to physical locations like s3://bucket/silver/sales/.

The curriculum covers the following technical components:

• The Metastore Abstraction: How logical schemas are decoupled from physical file paths.
• Partition Management: Mechanisms for partition discovery to ensure new data directories are queryable immediately upon arrival.
• Governance and Security: implementing Role-Based Access Control (RBAC) at the catalog level rather than the file level.
• Lineage Tracking: Methods for auditing the flow of data from raw sources to curated aggregates.

You will apply these concepts in a practical exercise by setting up a data catalog and configuring a crawler to generate table definitions from an existing dataset.