Modern deep learning compilers distinguish themselves from traditional language compilers by managing two distinct phases of abstraction. While a standard C++ compiler might lower source code directly into an instruction-level intermediate representation (IR) like LLVM IR, an AI compiler must first reason about the structure of the mathematics before it worries about the structure of the assembly code. This necessity gives rise to a multi-level IR architecture, split primarily into Graph-Level (High-Level) IR and Kernel-Level (Low-Level) IR.
The High-Level IR is the compiler's view of the model immediately after parsing the frontend framework code (such as PyTorch or TensorFlow). At this stage, the representation remains declarative. The compiler knows what needs to be computed but has not yet decided how to compute it.
In a Graph-Level IR, the fundamental unit of data is the Tensor. The compiler tracks properties such as data type (float32, int8), shape (dimensions), and memory layout (NCHW vs NHWC). The fundamental unit of execution is the Operator. A single node in this graph, such as Conv2D or Softmax, represents a massive amount of arithmetic operations.
Consider a typical layer in a transformer model. In High-Level IR, a fully connected layer followed by an activation function appears as a minimal set of nodes. The complexity of the underlying loops is hidden behind the operator abstraction.
A high-level graph representation of a dense layer. The nodes represent logical mathematical operations rather than CPU instructions.
The primary utility of this level is global optimization. Because the compiler sees the entire topology of the neural network, it can perform algebraic simplifications and structural changes. For example, if the compiler encounters the sequence:
It does not need to worry about register allocation or cache lines yet. It focuses on Operator Fusion, asking whether these three operations can be merged into a single kernel call to minimize memory round-trips to global memory (VRAM).
The transition from High-Level to Low-Level IR is often referred to as "lowering." This is a destructive process regarding semantics. Once a Conv2D node is lowered, the compiler loses the semantic knowledge that it is performing a convolution. Instead, the operation is replaced by its implementation details: nested loops, memory loads, multiply-accumulate instructions, and stores.
This distinction is important because certain optimizations are only possible at specific levels. You cannot effectively perform operator fusion once the code has been shattered into loops, as the structure is too obscured. Conversely, you cannot perform loop tiling or vectorization at the graph level because loops do not formally exist there yet.
Low-Level IR resembles a restricted subset of C or assembly language. It is imperative rather than declarative. At this stage, the abstraction shifts from Tensors to Buffers (blocks of allocated memory with pointers) and from Operators to Loop Nests.
In frameworks like Apache TVM (specifically the Tensor Intermediate Representation, or TIR) or MLIR's Affine/Linalg dialects, the Low-Level IR makes explicit everything that was implicit in the graph. The compiler must now manage:
Consider the same matrix multiplication from the graph example. In Low-Level IR, it is expanded into an iterative structure amenable to hardware-specific tuning.
Representation of the same operation after lowering. The focus shifts to iteration order, memory allocation, and scalar computation.
This level is where the "heavy lifting" of performance engineering occurs. The compiler attempts to reshape these loop nests to maximize data locality. Techniques such as Loop Tiling (breaking large loops into smaller blocks to fit in cache) and Vectorization (using SIMD instructions like AVX-512 or NEON) are applied here.
To architect an effective compiler pass, one must identify which IR level offers the correct primitives for the task. Attempting to perform memory planning at the graph level is imprecise because buffer lifespans are not fully determined. Attempting to perform dead code elimination at the loop level is inefficient compared to pruning the graph earlier.
The following breakdown illustrates the functional differences between these two abstraction layers.
Functional comparison distinguishing the scope and capabilities of graph-level versus kernel-level representations.
In modern infrastructures like MLIR (Multi-Level Intermediate Representation), the boundary between high and low levels is becoming less rigid. Instead of two monolithic states, the compiler may utilize a progressive lowering strategy involving several dialects.
For instance, the Linalg dialect in MLIR sits arguably in the middle. It represents operations like matrix multiplication effectively (retaining the semantic intent) but operates on buffers suitable for loop analysis. This allows the compiler to perform loop fusion, a technique usually reserved for loop-level IR, using graph-level information. This hybrid approach allows the compiler to make tiling decisions while still aware of the broader operator context, solving the "phase ordering" problem where optimizations at one level inadvertently pessimize the code for the next.
Understanding this dichotomy is essential for the practical session that follows. When inspecting TVM's Relay IR, you will see the declarative graph. As we move into later chapters on scheduling, you will see how that graph is lowered into the imperative TIR to be tuned for specific hardware constraints.
Was this section helpful?
© 2026 ApX Machine LearningEngineered with