How Dropout Works During Training

Dropout introduces randomness directly into the network's architecture during training. This method contrasts with regularization techniques that modify the loss function, like weight regularization. Instead, Dropout modifies the network itself on each forward pass.

The core idea is straightforward: for each training example (or mini-batch), temporarily remove units (along with their connections) from the network with a certain probability $p$. Each unit has an independent chance $p$ of being dropped, meaning its output is set to zero for that specific forward and backward pass.

The Dropout Mask

Imagine a layer in your network with $N$ units. During a training iteration's forward pass, we generate a random binary mask, let's call it $m$, of the same size as the layer's output. Each element $m_i$ in this mask is drawn from a Bernoulli distribution: it's 1 with probability $1-p$ (the "keep probability") and 0 with probability $p$ (the "dropout probability").

$$m_i \sim \text{Bernoulli}(1-p)$$

If a unit's corresponding mask value $m_i$ is 1, the unit operates as usual. If $m_i$ is 0, the unit's output is forced to zero for this specific forward pass. This mask is then applied element-wise to the layer's activation output, let's say $a$. The result, $a_{dropout}$, is then passed to the next layer.

$$a_{dropout} = a \odot m \quad (\text{where } \odot \text{ is element-wise multiplication})$$

Crucially, a different mask $m$ is generated for every training sample or mini-batch presented to the network. This constant "thinning" of the network prevents units from becoming overly specialized or dependent on the presence of specific other units.

A layer with four units undergoing Dropout during two different training steps. The grayed-out units with dashed outlines are randomly set to zero output (dropped) for that specific step. Notice the set of dropped units changes between steps.

Impact on Training

This process has a significant effect:

1. Forces Distributed Representations: Since any unit can disappear randomly, the network cannot afford to rely too heavily on any single unit. It learns more features that are somewhat redundant, distributed across multiple units. If one path is "dropped", others can potentially compensate.
2. Ensemble Effect (Approximation): Training a network with Dropout can be seen as implicitly training a large number of "thinned" networks that share weights. Each training step uses a different thinned network. While not a true ensemble average, this process provides similar regularization benefits.
3. Reduces Co-adaptation: Neurons become less sensitive to the specific noisy activations of other individual neurons, reducing complex co-adaptations that might only fit the training data.

Backward Pass Mechanics

During the backward pass (backpropagation), the gradients are only computed and propagated through the units that were not dropped (where $m_i = 1$). The units whose outputs were set to zero do not contribute to the gradient calculation for that particular training step, effectively being temporarily removed from the gradient computation path as well.

This constant shuffling and disabling of units during training is the primary mechanism by which Dropout helps prevent overfitting. However, it also means we need a consistent way to use the full network during inference or testing, which we will address in the next section on activation scaling.