Hands-on Practical: Adding Dropout Layers

Adding Dropout layers to a neural network applies this regularization technique. Integrating Dropout into a model using a framework like PyTorch is straightforward. How to add nn.Dropout layers will be demonstrated, along with a discussion of the implications for the training process.

Assume we have a simple Multi-Layer Perceptron (MLP) for a classification task. A potential architecture without Dropout might look like this:

import torch
import torch.nn as nn

class SimpleMLP(nn.Module):
    def __init__(self, input_size, hidden_size1, hidden_size2, output_size):
        super(SimpleMLP, self).__init__()
        self.layer_1 = nn.Linear(input_size, hidden_size1)
        self.relu_1 = nn.ReLU()
        self.layer_2 = nn.Linear(hidden_size1, hidden_size2)
        self.relu_2 = nn.ReLU()
        self.output_layer = nn.Linear(hidden_size2, output_size)

    def forward(self, x):
        x = self.layer_1(x)
        x = self.relu_1(x)
        x = self.layer_2(x)
        x = self.relu_2(x)
        x = self.output_layer(x)
        return x

# Example instantiation
# model_no_dropout = SimpleMLP(input_size=784, hidden_size1=256, hidden_size2=128, output_size=10)
# print(model_no_dropout)

This is a standard feedforward network. If this model were prone to overfitting on our dataset, we could introduce Dropout layers.

Adding Dropout Layers

The nn.Dropout module in PyTorch implements the Dropout technique. It takes the dropout probability p as an argument, which is the probability that any given neuron's output will be set to zero during training. A common practice is to place Dropout layers after the activation functions of the hidden layers.

Here's how we can modify our SimpleMLP to include Dropout:

import torch
import torch.nn as nn

class MLPWithDropout(nn.Module):
    def __init__(self, input_size, hidden_size1, hidden_size2, output_size, dropout_prob=0.5):
        super(MLPWithDropout, self).__init__()
        self.layer_1 = nn.Linear(input_size, hidden_size1)
        self.relu_1 = nn.ReLU()
        # Dropout after first hidden layer's activation
        self.dropout_1 = nn.Dropout(p=dropout_prob)
        self.layer_2 = nn.Linear(hidden_size1, hidden_size2)
        self.relu_2 = nn.ReLU()
        # Dropout after second hidden layer's activation
        self.dropout_2 = nn.Dropout(p=dropout_prob)
        self.output_layer = nn.Linear(hidden_size2, output_size)

    def forward(self, x):
        x = self.layer_1(x)
        x = self.relu_1(x)
        x = self.dropout_1(x) # Apply dropout
        x = self.layer_2(x)
        x = self.relu_2(x)
        x = self.dropout_2(x) # Apply dropout
        x = self.output_layer(x)
        return x

# Example instantiation with default dropout probability of 0.5
model_with_dropout = MLPWithDropout(input_size=784, hidden_size1=256, hidden_size2=128, output_size=10)
print(model_with_dropout)

# Or specify a different probability
# model_with_dropout_p25 = MLPWithDropout(input_size=784, hidden_size1=256, hidden_size2=128, output_size=10, dropout_prob=0.25)
# print(model_with_dropout_p25)

In this modified version:

1. We added a dropout_prob parameter to the constructor, defaulting to 0.5, a common starting value.
2. We instantiated nn.Dropout layers (self.dropout_1, self.dropout_2) with the specified probability.
3. In the forward method, we apply these Dropout layers immediately after the ReLU activations of the hidden layers. Note that Dropout is typically not applied to the output layer.

Training vs. Evaluation Mode

A significant aspect of using Dropout (and other layers like Batch Normalization) is the distinction between training and evaluation phases.

• During Training: We want Dropout to be active, randomly zeroing out neuron outputs to prevent co-adaptation.
• During Evaluation/Inference: We want to use the full network capacity. Dropout should be deactivated, and typically, the outputs of the layer preceding Dropout are scaled down by a factor of $(1-p)$ to account for the fact that more neurons are active than during training (this scaling is automatically handled by PyTorch's nn.Dropout when using model.eval(), implementing the "inverted dropout" technique).

PyTorch models have modes that handle this behavior. You must explicitly switch between them:

• model.train(): Sets the model to training mode. Dropout layers are active.
• model.eval(): Sets the model to evaluation mode. Dropout layers are inactive, and activations are scaled appropriately.

Here's a sketch of how this looks in a typical training loop:

# Assume model, train_loader, val_loader, optimizer, criterion are defined

num_epochs = 10

for epoch in range(num_epochs):
    # --- Training Phase ---
    model_with_dropout.train() # Set model to training mode
    train_loss = 0.0
    for data, target in train_loader: # Iterate over training batches
        optimizer.zero_grad()
        output = model_with_dropout(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        train_loss += loss.item() * data.size(0)
    train_loss /= len(train_loader.dataset)
    print(f"Epoch {epoch+1} Training Loss: {train_loss:.4f}")

    # --- Validation Phase ---
    model_with_dropout.eval() # Set model to evaluation mode
    val_loss = 0.0
    with torch.no_grad(): # Disable gradient calculations for validation
        for data, target in val_loader: # Iterate over validation batches
            output = model_with_dropout(data)
            loss = criterion(output, target)
            val_loss += loss.item() * data.size(0)
    val_loss /= len(val_loader.dataset)
    print(f"Epoch {epoch+1} Validation Loss: {val_loss:.4f}")

# --- Final Evaluation on Test Set ---
# model_with_dropout.eval() # Ensure model is in evaluation mode
# with torch.no_grad():
#    # Perform testing...

Forgetting to switch to model.eval() during validation or testing is a common mistake. It would lead to stochastic predictions (due to active Dropout) and incorrect performance measurements because the outputs aren't properly scaled.

Visualizing the Effect of Dropout

Dropout often helps bridge the gap between training performance and validation/test performance, indicating reduced overfitting. While training loss might be slightly higher or converge slower with Dropout (as the network effectively changes in each iteration), the validation loss should typically be lower and more stable compared to a model without Dropout that is overfitting.

Comparison of training and validation loss curves for models with and without Dropout. Note how the validation loss for the model without Dropout starts increasing (indicating overfitting), while the validation loss for the model with Dropout remains lower and more stable, although the training loss is slightly higher.

Experimenting Further

This practical demonstrates the basic implementation. You can now experiment with:

• Dropout Rate (p): Try different values (e.g., 0.2, 0.3, 0.5). Higher values provide stronger regularization but might slow down convergence or lead to underfitting if set too high.
• Placement: While placing Dropout after activations in fully connected layers is common, experiment with placing it before activations or only in specific layers.
• Combination: Combine Dropout with other regularization techniques like L2 weight decay, observing their joint effect.

Adding Dropout is a powerful tool in your arsenal against overfitting. Remember to use model.train() and model.eval() correctly to ensure it behaves as expected during different phases.