Compression in a data lake environment is not merely about saving disk space. While reducing storage costs is a primary benefit, the impact of compression on I/O throughput and query latency is often more significant. When a distributed query engine like Apache Spark or Trino reads data from object storage, the operation involves three distinct stages: reading the bytes from the network, decompressing those bytes in memory, and parsing the data into a usable format.
An uncompressed file transfers faster from the disk to the CPU if the network bandwidth is infinite, but cloud networks have limits. Conversely, a highly compressed file reduces network transfer time but demands significant CPU cycles to decompress. The goal of selecting a compression algorithm is to balance these opposing forces. You aim to minimize the total time defined as:
If the CPU takes longer to decompress the data than the network would have taken to transfer the raw bytes, the compression becomes a bottleneck.
Before examining specific algorithms, we must address the interaction between compression and distributed processing. Data lakes rely on the ability to split large files into smaller chunks (splits) that can be processed in parallel by different worker nodes.
Certain compression formats, such as a standard Gzip stream applied to a CSV file, are not splittable. To read the last record in a data.csv.gz file, the engine must decompress the entire stream from the beginning to locate the record boundaries. This forces a single CPU core to process the entire file, negating the benefits of distributed computing.
However, this limitation changes when using columnar formats like Apache Parquet or ORC. These formats divide data into internal "row groups" or blocks. The compression codec is applied to each block independently rather than the file as a whole. Therefore, a Parquet file using Gzip compression is fully splittable. The query engine reads the file metadata, identifies the byte ranges for each row group, and assigns different groups to different workers.
We focus on three primary codecs widely used in modern data architecture: Snappy, Gzip, and Zstandard (Zstd).
Snappy is designed for speed. It does not aim for maximum compression ratios but focuses on very high decompression speeds, often exceeding 500 MB/s on modern processors. It provides a reasonable reduction in size (typically 1.5x to 2x) with minimal CPU overhead.
Because of this balance, Snappy is the default compression codec for Apache Parquet. It is ideal for "hot" data layers where query latency is the priority. In interactive analytics where users expect responses in seconds, the low CPU cost of Snappy ensures that the processor spends its time executing query logic rather than decompressing bytes.
DEFLATE-based Gzip offers high compression ratios (often 3x to 4x for text data) but incurs a high CPU cost during both compression and decompression. In a data lake, Gzip is best suited for "cold" or archival data where storage costs are the primary concern and access is infrequent.
When used with text formats like JSON or CSV, Gzip creates non-splittable files. You should avoid generating large (multi-gigabyte) Gzip-compressed JSON files. Instead, if you must use Gzip with text, size the files to be roughly equal to your target partition size (e.g., 128MB - 256MB) so that the lack of internal splitting does not hinder parallelism.
Zstandard is a modern algorithm developed by Meta that provides a compression ratio comparable to Gzip but with decompression speeds closer to Snappy. It features a tunable compression level (ranging from negative values for speed to 22 for maximum compression).
Zstd has become the standard recommendation for general-purpose data lake storage. It offers a "pareto improvement" over Gzip, strictly dominating it in both speed and ratio at standard settings. Many organizations are migrating their Silver and Gold tables to Zstd-compressed Parquet to reduce storage costs without sacrificing query performance.
Comparison of typical decompression speeds versus compression ratios. Zstd occupies the favorable middle ground, offering high ratios with high speed.
A distinct advantage of columnar file formats is the ability to apply different encoding and compression strategies to different columns. Since data in a column is uniform (e.g., all integers or all timestamps), the compression algorithm can exploit this homogeneity.
For example, a column containing low-cardinality string data (like country_code) works exceptionally well with Dictionary Encoding combined with RLE (Run-Length Encoding), followed by general-purpose compression. This pipeline often results in storage footprints significantly smaller than row-oriented equivalents.
When configuring your ingestion jobs (using Spark or Flink), you generally define the compression codec at the table or file level. However, the underlying writer library (like parquet-mr) handles the application of encodings automatically based on the data type and statistics.
Choosing the right algorithm depends on the data lifecycle stage:
The following logic flow outlines the decision process for selecting a codec based on file format and access patterns.
Decision tree for selecting compression algorithms based on file format and performance requirements.
By correctly aligning the compression algorithm with the storage format and query requirements, you prevent the CPU from becoming a bottleneck in what should be an I/O-bound architecture.
Was this section helpful?
© 2026 ApX Machine LearningEngineered with