ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Model Regularization and Optimization in Deep Learning
Chapter 1: The Challenge of Generalization
Introduction to Model Generalization
Understanding Underfitting and Overfitting
The Bias-Variance Tradeoff in Deep Learning
Diagnosing Model Performance: Learning Curves
Validation and Cross-Validation Strategies
The Role of Regularization and Optimization
Setting up the Development Environment
Practice: Visualizing Overfitting
Quiz for Chapter 1
Chapter 2: Weight Regularization Techniques
Intuition Behind Weight Regularization
L2 Regularization (Weight Decay): Mechanism
L2 Regularization: Mathematical Formulation
L1 Regularization: Mechanism and Sparsity
L1 Regularization: Mathematical Formulation
Comparing L1 and L2 Regularization
Elastic Net: Combining L1 and L2
Implementing Weight Regularization
Hands-on Practical: Applying L1/L2 to a Network
Quiz for Chapter 2
Chapter 3: Dropout Regularization
Introducing Dropout: Preventing Co-adaptation
How Dropout Works During Training
Scaling Activations at Test Time
Inverted Dropout Implementation
Dropout Rate as a Hyperparameter
Considerations for Convolutional and Recurrent Layers
Implementing Dropout in Practice
Hands-on Practical: Adding Dropout Layers
Quiz for Chapter 3
Chapter 4: Normalization Techniques for Training Stability
The Problem of Internal Covariate Shift
Introduction to Batch Normalization
Batch Normalization: Forward Pass Calculation
Batch Normalization: Backward Pass Calculation
Benefits of Batch Normalization
Batch Normalization at Test Time
Considerations and Placement in Networks
Introduction to Layer Normalization
Implementing Batch Normalization
Hands-on Practical: Integrating Batch Normalization
Quiz for Chapter 4
Chapter 5: Foundational Optimization Algorithms
Revisiting Gradient Descent
Challenges with Standard Gradient Descent
Stochastic Gradient Descent (SGD)
Mini-batch Gradient Descent
SGD Challenges: Noise and Local Minima
SGD with Momentum: Accelerating Convergence
Nesterov Accelerated Gradient (NAG)
Implementing SGD and Momentum
Practice: Comparing GD, SGD, and Momentum
Quiz for Chapter 5
Chapter 6: Adaptive Optimization Algorithms
The Need for Adaptive Learning Rates
AdaGrad: Adapting Learning Rates per Parameter
AdaGrad Limitations: Diminishing Learning Rates
RMSprop: Addressing AdaGrad's Limitations
Adam: Adaptive Moment Estimation
Adam Algorithm Breakdown
Adamax and Nadam Variants (Brief Overview)
Choosing Between Optimizers: Guidelines
Implementing Adam and RMSprop
Hands-on Practical: Optimizer Comparison Experiment
Quiz for Chapter 6
Chapter 7: Optimization Refinements and Hyperparameter Tuning
Importance of Parameter Initialization
Common Initialization Strategies (Xavier, He)
Learning Rate Schedules: Motivation
Step Decay Schedules
Exponential Decay and Other Scheduling Methods
Warmup Strategies
Tuning Hyperparameters: Learning Rate, Regularization Strength, Batch Size
Relationship Between Batch Size and Learning Rate
Grid Search vs. Random Search for Hyperparameters
Implementing Learning Rate Scheduling
Practice: Tuning Hyperparameters for a Model
Quiz for Chapter 7
Chapter 8: Combining Techniques and Practical Considerations
Interaction Between Regularization and Optimization
Typical Deep Learning Training Workflow
Monitoring Training: Loss Curves and Metrics
Early Stopping as Regularization
Combining Dropout and Batch Normalization
Data Augmentation as Implicit Regularization
Choosing the Right Combination of Techniques
Debugging Training Issues Related to Optimization/Regularization
Hands-on Practical: Building and Tuning a Regularized/Optimized Model
Quiz for Chapter 8

Practice: Visualizing Overfitting

Generalization, overfitting, and underfitting are important concepts in deep learning. Learning curves serve as a diagnostic tool, indicating the gap between training performance and validation performance. To illustrate these ideas, a scenario is generated where overfitting is likely, and its effects are visualized directly. This demonstration will help in understanding how to spot overfitting using both performance metrics and the model's behavior.

We'll use a simple regression problem: fitting a model to data generated from a known function with added noise. By comparing a simple model to a complex one, we can observe how the complex model might learn the noise in the training data, leading to poor performance on unseen data.

Setting Up the Scenario

First, ensure you have your Python environment ready with PyTorch, NumPy, and Matplotlib (or another plotting library like Plotly or Seaborn) installed, as outlined in the "Setting up the Development Environment" section.

"We'll create a small synthetic dataset based on a simple underlying function, like a sine wave or a low-degree polynomial, and add some random noise. This noise mimics the imperfections and random variations present in data."

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt

# Generate synthetic data
torch.manual_seed(42) # for reproducibility
n_samples = 30
X = torch.linspace(-np.pi, np.pi, n_samples).unsqueeze(1)
# True function: sin(x) + small linear trend
y_true = torch.sin(X) + 0.1 * X
# Add noise
y = y_true + torch.randn(X.size()) * 0.2

# Split into training and validation sets (simple split for illustration)
n_train = 20
X_train, y_train = X[:n_train], y[:n_train]
X_val, y_val = X[n_train:], y[n_train:]

# Function to plot data
def plot_data(X_train, y_train, X_val, y_val, X_full=None, y_pred=None, model_name=None):
    plt.figure(figsize=(8, 5))
    plt.scatter(X_train.numpy(), y_train.numpy(), label='Training Data', c='#1f77b4', s=50, alpha=0.7)
    plt.scatter(X_val.numpy(), y_val.numpy(), label='Validation Data', c='#ff7f0e', s=50, alpha=0.7, marker='x')
    if X_full is not None and y_pred is not None:
        plt.plot(X_full.numpy(), y_pred.detach().numpy(), label=f'{model_name} Prediction', c='#2ca02c', linewidth=2)
    plt.xlabel("Feature (X)")
    plt.ylabel("Target (y)")
    plt.legend()
    plt.grid(True, linestyle='--', alpha=0.5)
    plt.title("Synthetic Data and Model Fit")
    plt.show()

# Plot the generated data
# plot_data(X_train, y_train, X_val, y_val) # Uncomment to see the raw data

Defining the Models

Now, let's define two neural network models using PyTorch. One will be relatively simple, and the other significantly more complex (with more layers or neurons), making it prone to overfitting on our small dataset.

# Simple Model (Potentially Underfitting or Good Fit)
class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        # A simple linear layer might underfit, let's try a small MLP
        self.layer1 = nn.Linear(1, 10)
        self.activation = nn.ReLU()
        self.output_layer = nn.Linear(10, 1)

    def forward(self, x):
        x = self.activation(self.layer1(x))
        x = self.output_layer(x)
        return x

# Complex Model (Prone to Overfitting)
class ComplexModel(nn.Module):
    def __init__(self):
        super().__init__()
        # More layers and neurons
        self.layer1 = nn.Linear(1, 128)
        self.activation1 = nn.ReLU()
        self.layer2 = nn.Linear(128, 128)
        self.activation2 = nn.ReLU()
        self.layer3 = nn.Linear(128, 64)
        self.activation3 = nn.ReLU()
        self.output_layer = nn.Linear(64, 1)

    def forward(self, x):
        x = self.activation1(self.layer1(x))
        x = self.activation2(self.layer2(x))
        x = self.activation3(self.layer3(x))
        x = self.output_layer(x)
        return x

simple_model = SimpleModel()
complex_model = ComplexModel()

Training the Models

We'll train both models using a standard training loop. We'll use Mean Squared Error (MSE) as the loss function and the Adam optimizer (which we'll cover in detail later, but it's a common default). We will track the training loss and validation loss for each epoch.

def train_model(model, X_train, y_train, X_val, y_val, epochs=2000, lr=0.005):
    criterion = nn.MSELoss()
    optimizer = optim.Adam(model.parameters(), lr=lr)

    train_losses = []
    val_losses = []

    for epoch in range(epochs):
        model.train() # Set model to training mode

        # Forward pass
        outputs = model(X_train)
        loss = criterion(outputs, y_train)

        # Backward pass and optimization
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # Evaluate on validation set
        model.eval() # Set model to evaluation mode
        with torch.no_grad():
            val_outputs = model(X_val)
            val_loss = criterion(val_outputs, y_val)

        train_losses.append(loss.item())
        val_losses.append(val_loss.item())

        if (epoch + 1) % 200 == 0:
            print(f'Epoch [{epoch+1}/{epochs}], Train Loss: {loss.item():.4f}, Val Loss: {val_loss.item():.4f}')

    return train_losses, val_losses

print("Training Simple Model...")
simple_train_losses, simple_val_losses = train_model(simple_model, X_train, y_train, X_val, y_val)

print("\nTraining Complex Model...")
complex_train_losses, complex_val_losses = train_model(complex_model, X_train, y_train, X_val, y_val)

Visualization 1: Learning Curves

Learning curves plot the training and validation loss over epochs. They are a primary tool for diagnosing fitting issues.

  • Good Fit: Training and validation loss decrease and converge.
  • Underfitting: Both losses remain high or plateau early.
  • Overfitting: Training loss continues to decrease while validation loss starts to increase or plateaus at a significantly higher level.

Let's plot the learning curves for both models.

Comparison of learning curves for the simple and complex models. Note the logarithmic scale on the y-axis. The complex model shows a clear divergence between training loss (decreasing) and validation loss (increasing), indicating overfitting. The simple model's losses converge more closely. (Note: Actual loss values depend on the specific run; these are illustrative.)

Observe the learning curves. The complex model's training loss likely drops very low, indicating it fits the training data well. However, its validation loss, after decreasing initially, probably starts to increase or levels off at a much higher value than the training loss. This widening gap is the classic sign of overfitting. The simple model might show less of a gap, or both losses might plateau, suggesting it's not complex enough to overfit (or perhaps even slightly underfits).

Visualization 2: Model Predictions

Another way to visualize overfitting, especially in regression, is to plot the model's predictions against the actual data points. An overfit model will often show excessive fluctuations, trying to pass through every training point, including the noise.

# Plot predictions
simple_model.eval()
complex_model.eval()

with torch.no_grad():
    # Use the full range X for plotting the curve
    y_pred_simple = simple_model(X) 
    y_pred_complex = complex_model(X)

# Plot Simple Model Fit
plot_data(X_train, y_train, X_val, y_val, X, y_pred_simple, "Simple Model")

# Plot Complex Model Fit
plot_data(X_train, y_train, X_val, y_val, X, y_pred_complex, "Complex Model")

# Optional: Plot true underlying function
plt.figure(figsize=(8, 5))
plt.scatter(X_train.numpy(), y_train.numpy(), label='Training Data', c='#1f77b4', s=50, alpha=0.7)
plt.scatter(X_val.numpy(), y_val.numpy(), label='Validation Data', c='#ff7f0e', s=50, alpha=0.7, marker='x')
plt.plot(X.numpy(), y_true.numpy(), label='True Function', c='black', linestyle=':', linewidth=2)
plt.plot(X.numpy(), y_pred_complex.detach().numpy(), label='Complex Model Prediction', c='#d62728', linewidth=2) # Use red for overfitting
plt.xlabel("Feature (X)")
plt.ylabel("Target (y)")
plt.legend()
plt.ylim(y.min()-0.5, y.max()+0.5) # Adjust y-limits for better view
plt.grid(True, linestyle='--', alpha=0.5)
plt.title("Complex Model vs. True Function")
plt.show()

When you examine the plots:

  1. Simple Model: The prediction curve should be relatively smooth. It might not capture all the nuances of the training data, but it likely represents the general trend without wild oscillations.
  2. Complex Model: This model's prediction curve might wiggle significantly, especially between training points. It tries too hard to accommodate every training data point, including the noise. This complex curve often leads to poor predictions for the validation points (the orange 'x' markers) and would likely perform poorly on any new data drawn from the true underlying function. The comparison with the true function (if plotted) highlights how far the overfitted model deviates.

Takeaways

This practice exercise demonstrated two visual methods for identifying overfitting:

  1. Learning Curves: Observing a divergence between training and validation loss over epochs.
  2. Model Predictions: Seeing the model fit the noise in the training data, resulting in an overly complex function that doesn't match the underlying pattern well.

Recognizing overfitting is the first step. The following chapters will introduce techniques like regularization (L1/L2, Dropout, Batch Normalization) and sophisticated optimization algorithms designed to combat this problem and help our models generalize better to unseen data.

© 2026 ApX Machine Learning