ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Introduction to Neural Networks
Chapter 1: Foundations of Neural Networks
From Biological to Artificial Neurons
Weights and Biases: The Network's Parameters
Activation Functions: Introducing Non-Linearity
Structuring Networks: Layers and Connections
A Simple Feedforward Network Example
Practice: Calculating Neuron Output
Quiz for Chapter 1
Chapter 2: Preparing Data for Neural Networks
Understanding Input Data Representation
Feature Scaling: Normalization and Standardization
Handling Categorical Data: Encoding Techniques
Creating Data Batches for Training
Splitting Data: Training, Validation, and Test Sets
Hands-on Practical: Preprocessing Sample Data
Quiz for Chapter 2
Chapter 3: Forward Propagation: Generating Predictions
The Flow of Information in a Network
Linear Transformation: Weighted Sum Calculation
Applying Activation Functions Layer-wise
Matrix Operations for Efficient Computation
Calculating the Final Output Prediction
Hands-on Practical: Implementing Forward Propagation
Quiz for Chapter 3
Chapter 4: Training: Backpropagation and Gradient Descent
Measuring Performance: Loss Functions
The Concept of Gradient Descent
Backpropagation: Calculating Gradients Efficiently
Updating Weights and Biases
Learning Rate and Its Importance
Stochastic Gradient Descent (SGD) and Variants
Practice: Calculating Gradients Manually
Quiz for Chapter 4
Chapter 5: Building and Training a Basic Neural Network
Setting up the Network Architecture
Initializing Weights and Biases
The Training Loop Structure
Implementing the Training Step
Monitoring Training Progress
Introduction to Deep Learning Frameworks (TensorFlow/PyTorch)
Hands-on Practical: Training a Simple Classifier
Quiz for Chapter 5
Chapter 6: Improving Network Performance and Generalization
Understanding Overfitting and Underfitting
The Role of Validation Sets
Regularization Techniques: L1 and L2
Dropout: Randomly Deactivating Neurons
Early Stopping: Halting Training Optimally
Hyperparameter Tuning Strategies
Practice: Applying Regularization
Quiz for Chapter 6

Backpropagation: Calculating Gradients Efficiently

Calculating a single gradient manually for a very simple network provides a clear illustration of the underlying mechanics. This direct approach helps in understanding the primary importance of the chain rule.

For this section: Don't worry about it if you don't understand.

Single gradient calculation

Let's take a single neuron with an input xxx, a weight www, a bias bbb, and an activation function σ\sigmaσ. The loss is LLL. The calculation is:

  1. Pre-activation: z=wx+bz = wx + bz=wx+b
  2. Activation: a=σ(z)a = \sigma(z)a=σ(z)
  3. Loss: L=Loss(a,y)L = \text{Loss}(a, y)L=Loss(a,y)

To find the gradient of the loss with respect to the weight, ∂L∂w\frac{\partial L}{\partial w}∂w∂L​, we need to see how a small change in www affects the loss LLL. The chain rule says we can find this by multiplying the rates of change at each step: ∂L∂w=∂L∂a×∂a∂z×∂z∂w\frac{\partial L}{\partial w} = \frac{\partial L}{\partial a} \times \frac{\partial a}{\partial z} \times \frac{\partial z}{\partial w}∂w∂L​=∂a∂L​×∂z∂a​×∂w∂z​ Let's break down each term:

  • ∂L∂a\frac{\partial L}{\partial a}∂a∂L​: This is how the loss changes when the neuron's output activation aaa changes. This is the first thing we'd calculate.
  • ∂a∂z\frac{\partial a}{\partial z}∂z∂a​: This is the derivative of the activation function, σ′(z)\sigma'(z)σ′(z). It tells us how the activation changes based on the pre-activation value.
  • ∂z∂w\frac{\partial z}{\partial w}∂w∂z​: This is the derivative of wx+bwx + bwx+b with respect to www, which is simply xxx.

So, to find the gradient for the weight www, we'd calculate each of these parts and multiply them. For a multi-layered network, this chain of derivatives would be much longer, connecting a parameter deep inside the network all the way to the final loss. The backpropagation algorithm is simply a structured and efficient way to perform this long chain rule calculation for all parameters at once.

The Power of the Chain Rule

Fundamentally, a neural network is a series of nested functions. The output of one layer becomes the input to the next. To find how a change in a parameter deep inside the network affects the final loss, we need to apply the chain rule repeatedly.

For example, a very simple network calculation:

  1. Input xxx
  2. Hidden layer pre-activation: z(1)=w(1)x+b(1)z^{(1)} = w^{(1)}x + b^{(1)}z(1)=w(1)x+b(1)
  3. Hidden layer activation: a(1)=σ(z(1))a^{(1)} = \sigma(z^{(1)})a(1)=σ(z(1)) (where σ\sigmaσ is an activation function)
  4. Output layer pre-activation: z(2)=w(2)a(1)+b(2)z^{(2)} = w^{(2)}a^{(1)} + b^{(2)}z(2)=w(2)a(1)+b(2)
  5. Output layer activation (prediction): a(2)=σ(z(2))a^{(2)} = \sigma(z^{(2)})a(2)=σ(z(2))
  6. Loss: L=Loss(a(2),y)L = \text{Loss}(a^{(2)}, y)L=Loss(a(2),y) (where yyy is the true label)

If we want to find the gradient of the loss LLL with respect to a weight in the first layer, say w(1)w^{(1)}w(1), we need to trace its influence forward to the loss: w(1)w^{(1)}w(1) affects z(1)z^{(1)}z(1), which affects a(1)a^{(1)}a(1), which affects z(2)z^{(2)}z(2), which affects a(2)a^{(2)}a(2), which finally affects LLL. The chain rule allows us to break this down:

∂L∂w(1)=∂L∂a(2)×∂a(2)∂z(2)×∂z(2)∂a(1)×∂a(1)∂z(1)×∂z(1)∂w(1)\frac{\partial L}{\partial w^{(1)}} = \frac{\partial L}{\partial a^{(2)}} \times \frac{\partial a^{(2)}}{\partial z^{(2)}} \times \frac{\partial z^{(2)}}{\partial a^{(1)}} \times \frac{\partial a^{(1)}}{\partial z^{(1)}} \times \frac{\partial z^{(1)}}{\partial w^{(1)}}∂w(1)∂L​=∂a(2)∂L​×∂z(2)∂a(2)​×∂a(1)∂z(2)​×∂z(1)∂a(1)​×∂w(1)∂z(1)​

Backpropagation organizes this calculation efficiently. It calculates terms like ∂L∂a(2)\frac{\partial L}{\partial a^{(2)}}∂a(2)∂L​ and ∂L∂z(2)\frac{\partial L}{\partial z^{(2)}}∂z(2)∂L​ first, and then reuses them to compute gradients for earlier layers.

The Backward Pass Explained

The algorithm works in two main phases:

  1. Forward Pass: Input data is fed through the network layer by layer, calculating pre-activations (z(l)z^{(l)}z(l)) and activations (a(l)a^{(l)}a(l)) at each step, ultimately producing the final output prediction a(L)a^{(L)}a(L). During this pass, we store the intermediate values (z(l)z^{(l)}z(l) and a(l)a^{(l)}a(l) for all layers lll) as they will be needed for the backward pass. The final loss LLL is calculated using the prediction a(L)a^{(L)}a(L) and the true target yyy.

  2. Backward Pass: This pass computes the gradients.

    • Output Layer: Calculate the gradient of the loss with respect to the output layer's activation, ∂L∂a(L)\frac{\partial L}{\partial a^{(L)}}∂a(L)∂L​. Then, using the chain rule, calculate the gradient with respect to the output layer's pre-activation z(L)z^{(L)}z(L): δ(L)=∂L∂z(L)=∂L∂a(L)∂a(L)∂z(L)=∂L∂a(L)σ′(z(L))\delta^{(L)} = \frac{\partial L}{\partial z^{(L)}} = \frac{\partial L}{\partial a^{(L)}} \frac{\partial a^{(L)}}{\partial z^{(L)}} = \frac{\partial L}{\partial a^{(L)}} \sigma'(z^{(L)})δ(L)=∂z(L)∂L​=∂a(L)∂L​∂z(L)∂a(L)​=∂a(L)∂L​σ′(z(L)) Here, σ′(z(L))\sigma'(z^{(L)})σ′(z(L)) is the derivative of the activation function used in the output layer, evaluated at z(L)z^{(L)}z(L). This term δ(L)\delta^{(L)}δ(L) represents the "error signal" at the output layer's pre-activation. Now, calculate the gradients for the output layer's weights W(L)W^{(L)}W(L) and biases b(L)b^{(L)}b(L): ∂L∂W(L)=∂L∂z(L)∂z(L)∂W(L)=δ(L)(a(L−1))T\frac{\partial L}{\partial W^{(L)}} = \frac{\partial L}{\partial z^{(L)}} \frac{\partial z^{(L)}}{\partial W^{(L)}} = \delta^{(L)} (a^{(L-1)})^T∂W(L)∂L​=∂z(L)∂L​∂W(L)∂z(L)​=δ(L)(a(L−1))T ∂L∂b(L)=∂L∂z(L)∂z(L)∂b(L)=δ(L)\frac{\partial L}{\partial b^{(L)}} = \frac{\partial L}{\partial z^{(L)}} \frac{\partial z^{(L)}}{\partial b^{(L)}} = \delta^{(L)}∂b(L)∂L​=∂z(L)∂L​∂b(L)∂z(L)​=δ(L) (Note: The exact forms depend on whether we are using vector/matrix notation. a(L−1)a^{(L-1)}a(L−1) represents the activations from the previous layer feeding into the output layer. The transpose ensures dimensions match for matrix multiplication).

    • Hidden Layers (working backward from l=L−1l = L-1l=L−1 down to l=1l=1l=1): For each hidden layer lll, calculate its error signal δ(l)\delta^{(l)}δ(l) based on the error signal δ(l+1)\delta^{(l+1)}δ(l+1) from the next layer (the one closer to the output): δ(l)=∂L∂z(l)=((W(l+1))Tδ(l+1))⊙σ′(z(l))\delta^{(l)} = \frac{\partial L}{\partial z^{(l)}} = \left( (W^{(l+1)})^T \delta^{(l+1)} \right) \odot \sigma'(z^{(l)})δ(l)=∂z(l)∂L​=((W(l+1))Tδ(l+1))⊙σ′(z(l)) Here, (W(l+1))T(W^{(l+1)})^T(W(l+1))T are the weights connecting layer lll to layer l+1l+1l+1. This step effectively propagates the error gradient backward through the network weights. The term σ′(z(l))\sigma'(z^{(l)})σ′(z(l)) is the derivative of the activation function of the current hidden layer lll. The symbol ⊙\odot⊙ denotes element-wise multiplication (Hadamard product). Once δ(l)\delta^{(l)}δ(l) is known, calculate the gradients for layer lll's weights W(l)W^{(l)}W(l) and biases b(l)b^{(l)}b(l) similar to the output layer: ∂L∂W(l)=δ(l)(a(l−1))T\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^T∂W(l)∂L​=δ(l)(a(l−1))T ∂L∂b(l)=δ(l)\frac{\partial L}{\partial b^{(l)}} = \delta^{(l)}∂b(l)∂L​=δ(l) Where a(l−1)a^{(l-1)}a(l−1) are the activations from the layer feeding into layer lll (for the first hidden layer, this would be the input data xxx).

This process continues backward until gradients for all parameters have been computed.

Visualizing the Flow

We can think of the computation as flowing forward to get the loss, and then backward to get the gradients.

A simplified view of forward (blue arrows) and backward (red arrows) passes. The forward pass computes activations and loss. The backward pass computes gradients, starting from the loss and propagating error signals (like δ(l)=∂L∂z(l)\delta^{(l)} = \frac{\partial L}{\partial z^{(l)}}δ(l)=∂z(l)∂L​) backward, reusing values calculated during the forward pass (a(l)a^{(l)}a(l), z(l)z^{(l)}z(l)) and network weights (W(l)W^{(l)}W(l)). Gradients for parameters (W,bW, bW,b) are derived from these error signals.

Why is it Efficient?

The efficiency of backpropagation comes from two main aspects:

  1. Reusing Calculations: Intermediate values (z(l)z^{(l)}z(l), a(l)a^{(l)}a(l)) computed during the forward pass are stored and reused during the backward pass. More importantly, the error signal (δ(l+1)\delta^{(l+1)}δ(l+1)) computed for layer l+1l+1l+1 is directly used to compute the error signal (δ(l)\delta^{(l)}δ(l)) for layer lll. This avoids redundant calculations.
  2. Dynamic Programming: It essentially uses dynamic programming. By calculating gradients layer by layer from back to front, it builds up the solution for the entire network's gradients without needing to recompute the influence paths for every single parameter independently.

Compared to numerically estimating gradients, which requires at least one extra forward pass for each parameter, backpropagation computes all gradients in roughly the same amount of computational time as a single forward pass. For a network with millions of parameters, this difference is enormous, making the training of deep networks feasible.

With the gradients computed via backpropagation, we now have the direction needed to update the weights and biases using gradient descent, which we will detail in the following sections.

  • Learning representations by back-propagating errors, David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, 1986 Nature, Vol. 323 DOI: 10.1038/323533a0 - This seminal paper popularized the backpropagation algorithm for training multi-layer neural networks.
  • Neural Networks and Deep Learning, Michael Nielsen, 2019 - An accessible online book providing an intuitive and step-by-step explanation of the backpropagation algorithm, suitable for beginners.
  • CS231n Convolutional Neural Networks for Visual Recognition: Optimization (Part 2), Andrej Karpathy, Justin Johnson, Fei-Fei Li (Stanford CS231n Course Staff), 2023 - Lecture notes offering a clear and concise explanation of backpropagation, emphasizing gradient flow and practical implementation details.

© 2026 ApX Machine Learning