Hands-on Practical: Profiling an Optimized Model

Theory is essential, but performance optimization is fundamentally an empirical science. Let's put the concepts from this chapter into practice by profiling a simple, optimized machine learning model component. We'll simulate a scenario where a matrix multiplication (GEMM), a fundamental operation in many ML models, has been compiled and optimized by an ML compiler targeting an NVIDIA GPU. Our goal is to use NVIDIA Nsight Compute to analyze its performance characteristics.

Scenario Setup

Assume we have a compiled executable, gemm_optimized, which performs a matrix multiplication $C = A \times B$, where $A$, $B$, and $C$ are large matrices. The ML compiler has applied optimizations like tiling, shared memory usage, and instruction scheduling to generate an efficient CUDA kernel.

Our target hardware is an NVIDIA GPU (e.g., an Ampere or Hopper architecture GPU). The primary tool we'll use is NVIDIA Nsight Compute (ncu), a detailed kernel profiler.

Step 1: Running the Profiler

Nsight Compute can be used from the command line or via its GUI. For automated analysis or scripting, the command line is often preferred. To capture a detailed profile, we can execute our compiled program under ncu.

# Ensure CUDA toolkit binaries are in your PATH
# Example: Profile the executable 'gemm_optimized'
# --set full: Collect a comprehensive set of metrics (can be time-consuming)
# -o profile_report: Save the report to a file named 'profile_report.ncu-rep'
# ./gemm_optimized: The executable to profile

ncu --set full -o profile_report ./gemm_optimized

This command runs gemm_optimized and gathers detailed performance data for every CUDA kernel launched by the application, saving it to profile_report.ncu-rep. For quicker analysis focused on specific aspects, you can use predefined metric sets (e.g., --set roofline, --set memory) or specify individual metrics.

Step 2: Analyzing the Profile Report

You can open the profile_report.ncu-rep file using the Nsight Compute GUI (nv-nsight-cu) or analyze aspects directly via the command line (ncu --query-metrics ... profile_report.ncu-rep). Let's focus on the key areas typically examined in the GUI or a detailed CLI report.

Identifying the Kernel

The report will list all CUDA kernels launched. Identify the primary kernel responsible for the matrix multiplication. It might be named something suggestive like gemm_kernel, matmul_core, or a mangled name derived from the compiler's internal representation. Focus your analysis on this kernel, especially if it consumes the majority of the GPU execution time.

Key Performance Sections

1. GPU Speed Of Light (SOL) Throughput:

   • This section provides a high-level view comparing the kernel's achieved computational throughput (FLOPS, Tensor Core operations) and memory bandwidth (DRAM, L2, L1) against the theoretical peak capabilities of the hardware.
   • Interpretation: A kernel operating near the compute peak is compute-bound. One near the memory bandwidth peak is memory-bound. Often, kernels fall somewhere in between, indicating potential bottlenecks in instruction latency, control flow, or cache utilization.

   This simplified roofline chart shows two hypothetical kernels. Kernel A (red) operates below both memory and compute roofs, suggesting potential inefficiencies. Kernel B (blue) is closer to the compute roof, indicating it's likely compute-bound.

2. Occupancy:

   • Measures how many warps (groups of 32 threads) are active on a Streaming Multiprocessor (SM) simultaneously, relative to the maximum possible. Low occupancy means the SM is underutilized, potentially hiding less latency.
   • Nsight Compute shows Achieved Occupancy and identifies limiters: Blocks per SM, Registers per Thread, Shared Memory per Block.
   • Interpretation: Low occupancy due to registers suggests register pressure (compiler might be spilling registers to local memory, which is slow). Low occupancy due to shared memory indicates the kernel configuration requests more shared memory per block than available, limiting concurrent blocks. Low occupancy due to block limits might mean the grid size is too small or the thread block size is too large for the problem.

3. Instruction Stats:

   • Provides insights into the efficiency of warp execution. Look at metrics like Issue Slot Utilization (how many instruction issue slots were used) and Executed Instructions per Clock (IPC).
   • Breakdowns by instruction type (integer, floating-point, memory, control flow) can reveal imbalances.
   • Interpretation: Low issue utilization or low IPC can point to instruction dependencies, long-latency operations (like sqrt or transcendental functions), or insufficient parallelism exposed by the compiler's scheduling. High control-flow divergence (different threads in a warp taking different paths) can significantly degrade performance.

4. Memory Workload Analysis:

   • Crucial for understanding memory-bound kernels. Examine L1/TEX Cache Hit Rate, L2 Cache Hit Rate, and DRAM Bandwidth.
   • Metrics like Memory Throughput show achieved bandwidth compared to peak. Look at the breakdown of memory operations (Global, Local, Shared).
   • Interpretation: Low cache hit rates combined with high DRAM bandwidth usage strongly indicate memory-boundness. This could stem from poor data locality (non-coalesced memory accesses) or an ineffective tiling strategy chosen by the compiler. High shared memory traffic should ideally correspond to high cache hit rates (if used as an L1 cache) or efficient reuse. High Local Memory traffic (spills) is generally undesirable.

   Comparison of achieved versus peak theoretical bandwidth at different memory levels. High utilization at the DRAM level suggests the kernel is likely memory-bound.

5. Source / Assembly Correlation (Optional but Powerful):

   • If source code correlation is set up during compilation and profiling, Nsight Compute can map performance counters back to specific lines of CUDA C++ or PTX/SASS assembly code.
   • Interpretation: This is invaluable for pinpointing exact instructions causing stalls (e.g., high-latency instructions, memory load/stores with poor cache performance). It helps verify if compiler optimizations like loop unrolling or instruction scheduling are effective or if specific generated instructions are bottlenecks.

Step 3: Iteration and Hypothesis Testing

Profiling is rarely a one-shot process. Based on the analysis:

• If compute-bound with low SOL: Check Instruction Stats for latency or scheduling issues. Examine assembly for inefficient instruction sequences. Perhaps the compiler needs hints or different schedule transformations (e.g., via pragmas or compiler flags if available).
• If memory-bound: Analyze Memory Workload details. Are cache hit rates low? Is DRAM bandwidth saturated? This might suggest tweaking tiling parameters, data layouts (if possible pre-compilation), or exploring different loop transformations in the compiler.
• If occupancy-limited: If limited by registers, can the kernel be simplified, or can the compiler be guided to use fewer registers? If limited by shared memory, can the algorithm be adapted, or the shared memory usage per block reduced?

This practical exercise demonstrates how profiling tools bridge the gap between high-level ML models and low-level hardware execution. By systematically analyzing performance metrics provided by tools like Nsight Compute, you can diagnose bottlenecks introduced or unresolved by the compiler and runtime, guiding further optimization efforts to achieve maximum performance for your ML workloads. Remember to consult the specific documentation for your chosen profiler and hardware for detailed metric definitions and advanced features.