ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Calculus Essentials for Machine Learning
Chapter 1: Introduction: Why Calculus Matters in Machine Learning
Functions and Models in ML
The Concept of Optimization in ML
Measuring Change: The Role of Derivatives
Calculus as a Tool for Understanding Algorithms
Quiz for Chapter 1
Chapter 2: Single-Variable Calculus: Derivatives and Optimization
Understanding Limits: The Foundation
Defining the Derivative
Common Differentiation Rules
Higher-Order Derivatives
Finding Minima and Maxima using Derivatives
Application: Simple Cost Function Optimization
Hands-on Practical: Calculating Derivatives with Python
Quiz for Chapter 2
Chapter 3: Multivariable Calculus: Gradients and Direction
Functions of Multiple Variables
Partial Derivatives
The Gradient Vector
Directional Derivatives
The Hessian Matrix
Multivariable Optimization Concepts
Hands-on Practical: Computing Gradients with NumPy
Quiz for Chapter 3
Chapter 4: Gradient Descent Algorithms
The Intuition Behind Gradient Descent
The Gradient Descent Algorithm Steps
The Learning Rate Parameter
Batch Gradient Descent
Stochastic Gradient Descent (SGD)
Mini-batch Gradient Descent
Challenges: Local Minima and Saddle Points
Hands-on Practical: Implementing Simple Gradient Descent
Quiz for Chapter 4
Chapter 5: The Chain Rule and Backpropagation
Revisiting the Chain Rule for Single Variables
The Chain Rule for Multivariable Functions
Introduction to Neural Networks as Composite Functions
Backpropagation: Applying the Chain Rule
Calculating Gradients for Weights and Biases
Computational Graphs
Hands-on Practical: Manual Backpropagation Example
Quiz for Chapter 5

Computational Graphs

Neural networks are essentially large, nested functions. Calculating the gradient of a loss function with respect to potentially millions of parameters deep inside this nested structure requires a systematic approach. Manually applying the chain rule repeatedly can quickly become overwhelming and error-prone; computational graphs provide a powerful and practical tool for this challenge.

A computational graph is a way to visualize a mathematical expression or a sequence of operations as a directed graph. Each node in the graph represents either a variable (input data, parameters, intermediate values) or an operation (like addition, multiplication, activation functions). The directed edges represent the flow of data and dependencies between these nodes; an edge from node A to node B means that the computation at node B depends on the output of node A.

Visualizing Computations

Let's look at a simple example. Suppose we have the function L=(x⋅w+b−y)2L = (x \cdot w + b - y)^2L=(x⋅w+b−y)2. This could represent a squared error loss for a very simple linear model z=x⋅w+bz = x \cdot w + bz=x⋅w+b with target yyy. We can break this down into elementary operations:

  1. p=x⋅wp = x \cdot wp=x⋅w (Multiplication)
  2. a=p+ba = p + ba=p+b (Addition)
  3. d=a−yd = a - yd=a−y (Subtraction)
  4. L=d2L = d^2L=d2 (Squaring)

We can represent this sequence as a computational graph:

A computational graph breaking down the calculation L=(x⋅w+b−y)2L = (x \cdot w + b - y)^2L=(x⋅w+b−y)2. Ellipses represent variables or computed values, while boxes represent operations. Edges show the direction of computation.

The Forward Pass

The forward pass involves computing the value of the expression by traversing the graph from the inputs to the final output. You start with the values of your inputs (like xxx, yyy) and parameters (like www, bbb), and then compute the value at each operation node based on the values of its parent nodes.

For our example, if x=2,w=3,b=1,y=5x=2, w=3, b=1, y=5x=2,w=3,b=1,y=5:

  1. p=x⋅w=2⋅3=6p = x \cdot w = 2 \cdot 3 = 6p=x⋅w=2⋅3=6
  2. a=p+b=6+1=7a = p + b = 6 + 1 = 7a=p+b=6+1=7
  3. d=a−y=7−5=2d = a - y = 7 - 5 = 2d=a−y=7−5=2
  4. L=d2=22=4L = d^2 = 2^2 = 4L=d2=22=4

The graph structure clearly defines the order of operations.

The Backward Pass: Backpropagation on the Graph

The real power of computational graphs becomes apparent during the backward pass, which is how backpropagation is implemented. The goal is to compute the gradient of the final output (our loss LLL) with respect to each input and parameter (i.e., ∂L∂x,∂L∂w,∂L∂b,∂L∂y\frac{\partial L}{\partial x}, \frac{\partial L}{\partial w}, \frac{\partial L}{\partial b}, \frac{\partial L}{\partial y}∂x∂L​,∂w∂L​,∂b∂L​,∂y∂L​).

We start at the final output node (LLL) and work backward through the graph. The gradient calculation relies on the chain rule applied locally at each node.

  1. Start at the end: The gradient of the output with respect to itself is always 1: ∂L∂L=1\frac{\partial L}{\partial L} = 1∂L∂L​=1. This is the initial gradient value we "feed" into the backward pass.

  2. Propagate gradients backward: For any node NNN that produces output zzz, if we know the gradient of the final loss LLL with respect to zzz (let's call this ∂L∂z\frac{\partial L}{\partial z}∂z∂L​), we can compute the gradient of LLL with respect to any input uuu of that node NNN using the chain rule: ∂L∂u=∂L∂z⋅∂z∂u\frac{\partial L}{\partial u} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial u}∂u∂L​=∂z∂L​⋅∂u∂z​ Here, ∂z∂u\frac{\partial z}{\partial u}∂u∂z​ is the local gradient of the operation at node NNN with respect to its specific input uuu.

  3. Sum gradients at forks: If a variable (like ppp in a more complex graph) feeds into multiple subsequent operations, its total gradient is the sum of the gradients flowing back from each path.

Let's trace this for our example graph:

  • Node sq (L=d2L=d^2L=d2):

    • Input gradient: ∂L∂L=1\frac{\partial L}{\partial L} = 1∂L∂L​=1.
    • Local gradient: ∂L∂d=∂(d2)∂d=2d\frac{\partial L}{\partial d} = \frac{\partial (d^2)}{\partial d} = 2d∂d∂L​=∂d∂(d2)​=2d. At our values, 2d=2(2)=42d = 2(2) = 42d=2(2)=4.
    • Output gradient (to node d): ∂L∂d=∂L∂L⋅∂L∂d=1⋅(2d)=2d=4\frac{\partial L}{\partial d} = \frac{\partial L}{\partial L} \cdot \frac{\partial L}{\partial d} = 1 \cdot (2d) = 2d = 4∂d∂L​=∂L∂L​⋅∂d∂L​=1⋅(2d)=2d=4.
  • Node sub (d=a−yd=a-yd=a−y):

    • Input gradient: ∂L∂d=4\frac{\partial L}{\partial d} = 4∂d∂L​=4.
    • Local gradients: ∂d∂a=1\frac{\partial d}{\partial a} = 1∂a∂d​=1, ∂d∂y=−1\frac{\partial d}{\partial y} = -1∂y∂d​=−1.
    • Output gradients:
      • To node a: ∂L∂a=∂L∂d⋅∂d∂a=4⋅1=4\frac{\partial L}{\partial a} = \frac{\partial L}{\partial d} \cdot \frac{\partial d}{\partial a} = 4 \cdot 1 = 4∂a∂L​=∂d∂L​⋅∂a∂d​=4⋅1=4.
      • To node y: ∂L∂y=∂L∂d⋅∂d∂y=4⋅(−1)=−4\frac{\partial L}{\partial y} = \frac{\partial L}{\partial d} \cdot \frac{\partial d}{\partial y} = 4 \cdot (-1) = -4∂y∂L​=∂d∂L​⋅∂y∂d​=4⋅(−1)=−4.
  • Node add (a=p+ba=p+ba=p+b):

    • Input gradient: ∂L∂a=4\frac{\partial L}{\partial a} = 4∂a∂L​=4.
    • Local gradients: ∂a∂p=1\frac{\partial a}{\partial p} = 1∂p∂a​=1, ∂a∂b=1\frac{\partial a}{\partial b} = 1∂b∂a​=1.
    • Output gradients:
      • To node p: ∂L∂p=∂L∂a⋅∂a∂p=4⋅1=4\frac{\partial L}{\partial p} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial p} = 4 \cdot 1 = 4∂p∂L​=∂a∂L​⋅∂p∂a​=4⋅1=4.
      • To node b: ∂L∂b=∂L∂a⋅∂a∂b=4⋅1=4\frac{\partial L}{\partial b} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial b} = 4 \cdot 1 = 4∂b∂L​=∂a∂L​⋅∂b∂a​=4⋅1=4.
  • Node mul (p=x⋅wp=x \cdot wp=x⋅w):

    • Input gradient: ∂L∂p=4\frac{\partial L}{\partial p} = 4∂p∂L​=4.
    • Local gradients: ∂p∂x=w\frac{\partial p}{\partial x} = w∂x∂p​=w, ∂p∂w=x\frac{\partial p}{\partial w} = x∂w∂p​=x. At our values, ∂p∂x=3\frac{\partial p}{\partial x} = 3∂x∂p​=3, ∂p∂w=2\frac{\partial p}{\partial w} = 2∂w∂p​=2.
    • Output gradients:
      • To node x: ∂L∂x=∂L∂p⋅∂p∂x=4⋅w=4⋅3=12\frac{\partial L}{\partial x} = \frac{\partial L}{\partial p} \cdot \frac{\partial p}{\partial x} = 4 \cdot w = 4 \cdot 3 = 12∂x∂L​=∂p∂L​⋅∂x∂p​=4⋅w=4⋅3=12.
      • To node w: ∂L∂w=∂L∂p⋅∂p∂w=4⋅x=4⋅2=8\frac{\partial L}{\partial w} = \frac{\partial L}{\partial p} \cdot \frac{\partial p}{\partial w} = 4 \cdot x = 4 \cdot 2 = 8∂w∂L​=∂p∂L​⋅∂w∂p​=4⋅x=4⋅2=8.

We have now computed all the required gradients: ∂L∂x=12\frac{\partial L}{\partial x} = 12∂x∂L​=12, ∂L∂w=8\frac{\partial L}{\partial w} = 8∂w∂L​=8, ∂L∂b=4\frac{\partial L}{\partial b} = 4∂b∂L​=4, and ∂L∂y=−4\frac{\partial L}{\partial y} = -4∂y∂L​=−4. The graph structure allowed us to systematically apply the chain rule without getting lost in the nested function structure.

Relevance to Deep Learning Frameworks

Modern deep learning libraries like TensorFlow and PyTorch heavily rely on this concept. They automatically build computational graphs (either statically before execution or dynamically during execution) based on the operations you define in your model code. This graph representation allows them to:

  1. Efficiently compute the forward pass.
  2. Automatically compute gradients for all parameters using backpropagation (often called "autograd" or automatic differentiation). This frees the developer from manually deriving and implementing the gradient calculations, which is essential for complex architectures.
  3. Optimize computations, for example, by identifying parallelizable operations or simplifying parts of the graph.

Understanding computational graphs provides insight into the mechanics behind these powerful frameworks and clarifies how the chain rule enables the training of even very deep neural networks. It turns the potentially complex process of gradient calculation into a structured traversal of a graph.

  • Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - Offers a comprehensive theoretical foundation for computational graphs and backpropagation within deep learning.
  • PyTorch Autograd: Automatic Differentiation, PyTorch Development Team, 2025 (PyTorch Foundation) - Explains how automatic differentiation is implemented using dynamic computational graphs in PyTorch.
  • CS231n: Deep Learning for Computer Vision, Lecture Notes on Backpropagation, Andrej Karpathy, 2023 - Provides clear explanations and visualizations of backpropagation and computational graphs in the context of neural networks.

© 2026 ApX Machine Learning