Mini-batch Gradient Descent

In the previous sections, we explored two extremes for computing the gradient: using the entire dataset (Batch Gradient Descent) and using just a single example (Stochastic Gradient Descent). Batch Gradient Descent provides an accurate estimate of the gradient but can be computationally expensive and slow for large datasets. Stochastic Gradient Descent updates parameters very frequently, leading to faster iterations, but the updates are noisy and the convergence path can oscillate significantly.

Mini-batch Gradient Descent offers a practical and widely used compromise between these two approaches. Instead of using all $m$ training examples or just one, it computes the gradient and updates the parameters using a small, randomly selected subset of the training data called a "mini-batch".

The Mechanics of Mini-batch Gradient Descent

The core idea is to process the data in small batches. Let the size of a mini-batch be denoted by $b$. Common mini-batch sizes are powers of 2, such as 32, 64, 128, or 256, chosen often for hardware optimization reasons, but the optimal size can depend on the specific problem and dataset.

The algorithm proceeds as follows:

1. Shuffle: At the beginning of each epoch (one full pass through the training dataset), shuffle the entire training dataset randomly. This helps prevent the model from being biased by the order of examples and improves convergence.
2. Partition: Divide the shuffled dataset into mini-batches of size $b$. If the total number of training examples $m$ is not perfectly divisible by $b$, the last mini-batch might be smaller.
3. Iterate: For each mini-batch $B$ (containing $b$ examples):
   • Compute the cost function $J_{B}(\theta)$ averaged over the examples in the mini-batch $B$.
   • Calculate the gradient $\nabla J_{B}(\theta)$ using only the examples in the current mini-batch $B$.
   • Update the model parameters $\theta$ using the gradient descent update rule: $$\theta := \theta - \alpha \nabla J_{B}(\theta)$$ where $\alpha$ is the learning rate.

Notice that the gradient $\nabla J_{B}(\theta)$ is an estimate of the true gradient $\nabla J(\theta)$ (the gradient calculated over the full dataset). It's less noisy than the SGD gradient (based on one example) but less accurate than the Batch GD gradient (based on all examples).

Advantages of Mini-batch Gradient Descent

Mini-batch Gradient Descent strikes a beneficial balance:

• Computational Efficiency: Calculating the gradient for a mini-batch is significantly faster than for the entire dataset, allowing for more frequent parameter updates compared to Batch GD. This often leads to faster overall convergence.
• Vectorization: Modern computing libraries (like NumPy, TensorFlow, PyTorch) are highly optimized for matrix and vector operations. Mini-batch processing allows us to leverage these optimized routines (vectorization) by performing calculations on the entire mini-batch simultaneously, making the computation much more efficient than processing examples one-by-one as in SGD.
• Smoother Convergence: The average gradient over a mini-batch provides a better estimate of the true gradient compared to SGD. This results in less noisy updates and a more stable convergence path, reducing the oscillations often seen with SGD. While not perfectly smooth like Batch GD, it generally moves more consistently towards the minimum.

Choosing the Mini-batch Size

The mini-batch size $b$ is a hyperparameter that needs to be chosen.

• Small $b$ (e.g., 1): Approaches SGD, resulting in noisy updates but frequent progress.
• Large $b$ (e.g., $m$): Becomes Batch GD, providing accurate gradients but slow updates.
• Intermediate $b$ (e.g., 32 to 512): Offers a balance. It leverages vectorization efficiencies and provides a reasonably good gradient estimate for stable updates.

The optimal mini-batch size often depends on the dataset size, model complexity, and hardware characteristics. It's typically found through experimentation.

Summary of Trade-offs

Feature	Batch GD	Stochastic GD (SGD)	Mini-batch GD
Data per update	Entire dataset ($m$)	Single example (1)	Mini-batch ($b$)
Update Frequency	Low (once per epoch)	High ($m$ per epoch)	Medium ($m/b$ per epoch)
Gradient Quality	Exact (True Gradient)	Noisy (High Variance)	Estimate (Reduced Variance)
Computation Cost	High per update	Low per update	Medium per update
Convergence	Smooth, potentially slow	Noisy, potentially fast	Relatively smooth & fast
Vectorization	Yes	Less effective	Yes (Effective)
Memory Usage	Can be high	Low	Moderate

Because it balances computational efficiency, update frequency, and convergence stability, Mini-batch Gradient Descent is the most commonly used optimization algorithm for training machine learning models, especially deep neural networks, on large datasets. It provides a practical way to navigate the cost function landscape effectively.