While profilers provide detailed runtime measurements of what is happening during execution, they don't always explain why. Compiler optimization reports offer a complementary perspective, providing explanations of the decisions made during the compilation process itself. Understanding these reports is a necessary skill for diagnosing performance issues that stem from the compiler's transformation strategy.
ML compilers, such as XLA, TVM, Glow, or vendor-specific tools, often generate verbose logs when instructed. These logs document the sequence of optimization passes applied, the specific transformations performed on the Intermediate Representation (IR), and sometimes, the reasons why certain optimizations were not applied. Sifting through this information can reveal important details about how your high-level model graph was translated into low-level, hardware-specific code.
What to Look For in Optimization Reports
The structure and content of optimization reports vary significantly between different compilers. However, common elements often include:
- Optimization Pass Sequencing: A list of the optimization passes executed, often in order. This helps understand the transformation pipeline.
- Transformation Logs: Details about specific changes made to the IR. This might include nodes fused, loops tiled, data layouts changed, or algebraic simplifications applied.
- Diagnostic Messages: Warnings or informational messages about potential issues or reasons for optimization decisions. For instance, a report might explain why two operations couldn't be fused due to incompatible data layouts or dependencies.
- Resource Allocation Information: Notes on estimated register pressure, memory usage calculations for static planning, or scheduling decisions for parallel execution.
- Kernel Generation Details: Information about the generated kernels, possibly including their names or signatures, target hardware features used (like Tensor Cores or specific SIMD instructions), and chosen configurations (thread block sizes, tile sizes).
- IR Snapshots (Optional): Some compilers allow dumping the IR at various stages, which can be invaluable for detailed analysis when correlated with the report logs.
Correlating Reports with Optimization Goals
Let's examine how reports can illuminate specific optimization outcomes:
Graph-Level Optimizations:
- Operator Fusion: Reports often explicitly log fusion decisions. You might see messages like "Fusing op
Conv2D_1 with BiasAdd_1 and Relu_1 into FusedConv2DBiasRelu_1". Conversely, you might find messages explaining why fusion failed, e.g., "Cannot fuse OpA and OpB: mismatched dimensions or intervening dependency". This directly correlates with profiler observations of kernel launch overhead; fewer, larger fused kernels should reduce this overhead.
- Layout Transformations: Logs may indicate changes like "Transforming tensor
T1 from NCHW to NHWC for target GPU_XYZ" or "Inserting layout transpose for OpC". If profiling shows high latency in memory-bound operations, checking the layout decisions in the report is a good starting point. Did the compiler choose the optimal layout for the sequence of operations and the target hardware?
A compiler report might log the transformation of separate Conv2D, BiasAdd, and ReLU operations into a single fused kernel.
Tensor-Level Optimizations:
- Tiling and Loop Transformations: Polyhedral compilers or loop optimizers often report the chosen tile sizes, loop permutations, or skewing factors. Messages might look like "Tiling loop nest for
MatMul_1 with tile sizes [64, 64, 16]" or "Applying loop permutation (i, j, k) -> (j, k, i)". If a kernel shows poor cache utilization or insufficient parallelism in the profiler, examining the tiling strategy in the report can reveal if suboptimal choices were made.
- Vectorization: Reports may indicate successful vectorization ("Vectorizing inner loop of
Kernel_X using AVX512") or reasons for failure ("Cannot vectorize loop: presence of non-vectorizable instructions or complex control flow"). This helps interpret CPU utilization and instruction mix reported by profilers like VTune.
Quantization:
- Reports are essential for understanding how quantization was implemented. Look for logs detailing:
- Insertion points of quantize/dequantize operations.
- Chosen scaling factors and zero points (especially for PTQ).
- Mapping of operations to low-precision kernels (e.g., "Using INT8 GEMM kernel for
QuantizedMatMul_5").
- Fallback operations: Which ops remained in FP32 or FP16 if mixed-precision was used.
Bridging Reports and Profiler Data
The real power comes from combining profiler insights with compiler reports.
- Profile First: Identify the most time-consuming kernels or operations and their characteristics (compute-bound, memory-bound, high latency).
- Consult the Report: Search the compiler logs for entries related to these specific operations or the optimizations relevant to the observed bottleneck.
- High Kernel Launch Overhead? Check the report for fusion activity. Was it successful? Were there many small, unfused kernels left?
- Low Arithmetic Intensity / Memory Bound? Examine reports for tiling, prefetching, and data layout decisions. Was data movement optimized?
- Underutilization of Compute Units (e.g., low FLOPs)? Look for vectorization reports, thread mapping strategies (for GPUs), or use of specialized units (Tensor Cores).
- Accuracy Issues with Quantized Models? Check the report for which layers were quantized, the parameters used, and whether QAT-specific passes ran correctly.
Challenges and Best Practices
Interpreting compiler reports isn't always straightforward:
- Verbosity: Reports can be extremely long and detailed. Use tools like
grep, awk, or custom scripts to filter relevant information. Compiler flags often exist to control verbosity or log specific passes (e.g., TF_XLA_FLAGS=--xla_dump_to=/path/, TVM's PassContext logging).
- Compiler-Specific Jargon: Each compiler uses its own terminology and IR naming conventions. Familiarity with the specific compiler's documentation and internal workings is often required.
- Mapping to Source: Connecting a log entry about an optimized internal IR node (e.g.,
HLO_Fusion_123) back to the original TensorFlow or PyTorch operation requires understanding the compiler's naming schemes or using debugging information if available.
- "Silent Failures": Sometimes, an optimization is not applied, but the report doesn't explicitly state why. This often requires deeper analysis, potentially involving dumping and inspecting the IR before and after the expected pass.
Despite these challenges, actively using compiler reports is an indispensable part of the expert ML performance engineer's toolkit. They provide the "why" behind the "what" observed in profilers, enabling more targeted optimization efforts, better compiler flag tuning, and sometimes even informing beneficial changes to the original model architecture itself.
