The RealNVP Architecture

The RealNVP architecture, which stands for Real-valued Non-Volume Preserving transformations, scales affine coupling layers to high-dimensional datasets like images. Instead of simply slicing a one-dimensional array in half, RealNVP introduces structured ways to partition multi-dimensional data using binary masks.

When dealing with spatial data such as images, explicitly splitting a tensor into two separate tensors becomes complex and computationally inefficient. RealNVP simplifies this by applying a binary mask. A binary mask is a tensor of the exact same shape as the input, filled entirely with zeros and ones. We use this mask to explicitly define which features pass through the layer unchanged and which features undergo the affine transformation.

Let $b$ represent this binary mask. We can rewrite the standard affine coupling operation using element-wise multiplication.

$y = b \odot x + (1 - b) \odot (x \odot \exp(s(b \odot x)) + t(b \odot x))$

In this equation, the elements where $b=1$ remain exactly as they were in the input $x$. The elements where $b=0$ are modified by the scaling function $s$ and the translation function $t$. Because $s$ and $t$ only receive the unmodified parts of the input, the Jacobian matrix remains triangular, allowing for fast determinant calculation and inversion.

To effectively capture the relationships between different pixels in an image, the authors of RealNVP proposed two specific masking patterns: checkerboard masking and channel-wise masking.

In checkerboard masking, the spatial dimensions of the image are masked in an alternating pattern that resembles a standard chess board. A pixel assigned a mask value of 1 is directly adjacent to pixels with a mask value of 0. This arrangement forces the model to use the immediate spatial context surrounding a pixel to predict its transformation parameters.

In channel-wise masking, the spatial dimensions remain entirely intact. The split occurs along the depth dimension of the image tensor. If a specific intermediate tensor contains 64 channels, the first 32 channels are assigned a mask value of 1, and the remaining 32 channels are assigned a mask value of 0.

A single coupling layer only updates a fraction of the input data. If you use the exact same mask for every layer in your network, the elements assigned a 1 will never be transformed. To solve this, RealNVP architectures stack multiple coupling layers and reverse the binary mask between operations. If a pixel passes through unchanged in the first layer, the inverted mask in the second layer ensures it gets transformed, using the newly updated pixels as its conditioning context.

Alternating mask operation across two consecutive coupling layers to ensure all input variables are eventually transformed.

The functions $s$ and $t$ are typically implemented as deep convolutional neural networks, often utilizing residual blocks. An important property of the RealNVP architecture is that these internal networks do not need to be invertible themselves. The invertibility of the entire model is mathematically guaranteed by the coupling architecture alone. This allows you to use standard, highly expressive neural network layers for $s$ and $t$ without worrying about their individual mathematical properties.

When training these architectures, numerical stability requires careful attention. The affine coupling equation relies on exponentiating the output of the scaling network, $\exp(s(x))$. If the network $s$ outputs a large positive number, the exponential function will explode, causing gradient calculations to fail. Implementations often apply a tanh activation function to the output of $s$, which constrains the values between -1 and 1. This constrained output is then scaled by a learnable parameter to allow the model sufficient flexibility while preventing explosive numerical values during maximum likelihood estimation.