CPU Performance Analysis (VTune, perf)

While GPUs and specialized accelerators often handle the bulk of the computation in ML workloads, the CPU remains a significant component. It executes parts of the model graph, drives the accelerator, manages the runtime system, handles data loading and preprocessing, and runs control flow logic. Optimizing CPU performance is therefore essential for overall application speed. Details are provided on how to use specialized CPU profiling tools, specifically Intel VTune Profiler and Linux perf, to dissect the performance of compiled ML code running on the CPU.

As highlighted in the chapter introduction, profiling code transformed by ML compilers presents unique challenges. Function names might be mangled or correspond to large, fused kernels, making direct correlation to the original model difficult. Furthermore, the heavy use of libraries (like MKL-DNN/oneDNN, OpenBLAS) means performance often depends on these pre-optimized routines. Effective CPU profiling requires tools that can look past the source code level and analyze the actual hardware execution details.

Intel VTune Profiler

Intel VTune Profiler is a powerful performance analysis tool for understanding CPU (and GPU/FPGA) behavior. It provides a graphical interface and multiple analysis types suitable for exploring ML workloads.

Analysis Types for ML

• Hotspots Analysis: This is often the starting point. It identifies the functions or code regions where the most CPU time is spent using low-overhead sampling. For compiled ML code, this often points to specific compute kernels (generated or from libraries), runtime scheduling functions, or memory management routines. It helps prioritize optimization efforts.
• Microarchitecture Exploration: This analysis goes deeper, using hardware performance monitoring units (PMUs) to collect detailed metrics about the CPU pipeline. You can analyze metrics like:
  • Clockticks per Instruction Retired (CPI): A high CPI indicates inefficiency; the CPU is spending many cycles per useful instruction.
  • Cache Misses: Identify high rates of L1, L2, or L3 cache misses, which stall the CPU pipeline. This often points to poor data locality, potentially addressable by compiler tiling strategies or layout transformations.
  • Branch Misprediction Rate: High rates indicate problematic control flow, which can sometimes be optimized by the compiler or runtime.
  • Vectorization Intensity: Measures how effectively SIMD (Single Instruction, Multiple Data) units (like AVX2, AVX-512) are being utilized. Low intensity in compute-bound kernels suggests missed vectorization opportunities by the compiler.
• Memory Access Analysis: Important for tensor-heavy workloads. This analysis helps pinpoint memory bandwidth bottlenecks, distinguish between cache-bound and DRAM-bound operations, and analyze memory latency issues. It can reveal inefficient memory access patterns generated by the compiler's loop transformations or data layout choices.

Using VTune for Compiled ML

You typically run VTune by either launching your ML application under its control or attaching it to an already running process. VTune collects data and presents it in various views (e.g., summary, bottom-up, caller/callee). You can often drill down from function-level summaries to source code or assembly, although correlating highly optimized, generated code back to the original ML graph operation requires understanding the compiler's transformations. Look for annotations showing PMU event counts directly on assembly instructions for fine-grained analysis.

Example breakdown of CPU time from a VTune Microarchitecture Exploration, showing potential bottlenecks like memory stalls (L1/L2/L3 Bound) or instruction starvation (Frontend Bound) versus useful work (Retiring). KernelA_Fused shows significant time potentially stalled on memory access.

Linux perf

Linux perf is a powerful, versatile command-line profiling tool built into the Linux kernel. It uses the CPU's Performance Monitoring Units (PMUs) to sample or count hardware events with low overhead.

Common perf Usage

• perf stat: Provides aggregate counts for common hardware events (cycles, instructions, cache misses, branch misses) for a command or process ID. Useful for a quick overview of performance characteristics.

  # Count events for the entire execution of an ML inference script
  perf stat python run_inference.py --model compiled_model.bin
  

• perf record: Samples the program's execution. It periodically records the instruction pointer and other information (like the call stack using the -g option). This generates a perf.data file.

  # Record performance data with call graphs
  perf record -g ./my_compiled_ml_app --input data.npy
  

• perf report: Analyzes the perf.data file generated by perf record. It presents a hierarchical view showing the percentage of samples collected in each function, library, or even individual instruction. You can navigate this report interactively in the terminal to explore hotspots and call chains.

• perf annotate: Disassembles functions identified as hotspots by perf report and annotates the assembly code with the percentage of samples that occurred at each instruction. This is invaluable for pinpointing the exact instructions causing stalls or consuming cycles, especially within compiler-generated code.

perf Strengths and Weaknesses

• Strengths: Lightweight, ubiquitous on Linux systems, kernel-level visibility, highly scriptable, supports an extensive number of hardware and software events.
• Weaknesses: Primarily command-line based (though external GUIs exist), interpretation can be less intuitive than VTune for complex microarchitectural issues, correlating samples precisely back to high-level source code in heavily optimized binaries can sometimes be challenging without proper debug symbols.

Using perf for Compiled ML

Similar to VTune, you use perf record to capture data while your compiled model executes. perf report will often show hotspots within the ML runtime's execution engine, specific kernels generated by the ML compiler, or within vendor libraries like oneDNN or OpenBLAS. Using perf annotate on these hotspot functions allows you to examine the assembly code and see where hardware events (like cache misses recorded via perf record -e cache-misses ...) are occurring frequently. This can directly inform compiler development or tuning. For instance, seeing numerous cache misses on load instructions within a generated loop nest might suggest that the compiler's tiling or prefetching strategy needs adjustment.

Interpretation and Action

Both VTune and perf provide data; the skill lies in interpreting it within the context of ML compilation.

• High CPI / Frontend Bound: May indicate instruction fetch bottlenecks or complex dependencies. Compiler instruction scheduling or code layout optimizations might help.
• High Cache Miss Rates / Memory Bound: Suggests poor data locality. This points towards improving compiler tiling strategies, data layout transformations (e.g., NCHW vs NHWC), or software prefetching.
• Low Vectorization Intensity: Indicates the compiler failed to effectively use SIMD units. This might require changes to loop transformations, data alignment, or explicit use of vector intrinsics during code generation.
• Runtime Overhead: If significant time is spent in runtime functions (scheduling, memory allocation), it indicates bottlenecks in the runtime system design itself, perhaps requiring optimization of the scheduler or memory manager.

By systematically applying these CPU profiling tools, you can move past guesswork and obtain concrete data about how your compiled ML code interacts with the hardware. This data is essential for diagnosing performance limitations and guiding the sophisticated compiler and runtime optimizations discussed throughout this course.