Apache Parquet Internals

Was this section helpful?

References

Apache Parquet Format Specification, Apache Parquet Project, 2024 - Provides the definitive technical specification of the Apache Parquet file format, detailing its internal structure, metadata, encoding, and data types.
C-Store: A Column-Oriented DBMS, Michael Stonebraker, Daniel J. Abadi, Adam Batkin, Xuedong Chen, Mitch Cherniack, Miguel Ferreira, Edmond Lau, Amerson Lin, Samuel R. Madden, Elizabeth J. O'Neil, Patrick E. O'Neil, Alexander Rasin, Nga Tran, Stan B. Zdonik, 2005 Proceedings of the 31st International Conference on Very Large Data Bases (VLDB) (VLDB Endowment) DOI: 10.1007/11566322_38 - Describes the foundational concepts of column-oriented database systems, explaining the principles behind their efficiency for analytical workloads, which are directly applied in formats like Parquet.
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017 (O'Reilly Media) - Chapter 3, 'Storage and Retrieval', includes sections on columnar storage and encoding, providing a conceptual framework that helps understand Parquet's design choices for analytical workloads.