ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Introduction to Deep Learning
Chapter 1: Neural Network Foundations
From Machine Learning to Deep Learning
Biological Inspiration: The Neuron
The Artificial Neuron: A Mathematical Model
The Perceptron: The Simplest Neural Network
Limitations of Single-Layer Perceptrons
Multi-Layer Perceptrons (MLPs): Adding Depth
Hands-on Practical: Building a Simple Perceptron Model
Quiz for Chapter 1
Chapter 2: Activation Functions and Network Architecture
The Role of Activation Functions
Sigmoid Activation
Hyperbolic Tangent (Tanh) Activation
Rectified Linear Unit (ReLU)
Variants of ReLU (Leaky ReLU, PReLU, ELU)
Choosing the Right Activation Function
Understanding Network Layers: Input, Hidden, Output
Designing Feedforward Network Architectures
Hands-on Practical: Implementing Different Activations
Quiz for Chapter 2
Chapter 3: Training Neural Networks: Loss and Optimization
Measuring Performance: Loss Functions
Common Loss Functions for Regression (MSE, MAE)
Common Loss Functions for Classification (Cross-Entropy)
Optimization: Finding the Best Weights
Gradient Descent Algorithm
Learning Rate
Stochastic Gradient Descent (SGD)
Challenges with Gradient Descent
Hands-on Practical: Visualizing Gradient Descent
Quiz for Chapter 3
Chapter 4: Backpropagation and Advanced Optimization
Calculating Gradients: The Chain Rule
Computational Graphs
The Backpropagation Algorithm Explained
Forward Pass vs. Backward Pass
Gradient Descent with Momentum
RMSprop Optimizer
Adam Optimizer
Choosing an Optimization Algorithm
Hands-on Practical: Backpropagation Step-by-Step
Quiz for Chapter 4
Chapter 5: Building and Training Deep Neural Networks
Introduction to Deep Learning Frameworks (TensorFlow/Keras, PyTorch)
Setting up the Development Environment
Preparing Data for Neural Networks
Defining a Feedforward Network Model
Weight Initialization Strategies
Compiling the Model: Loss and Optimizer Selection
Training the Model: The fit Method
Monitoring Training Progress (Loss and Metrics)
Evaluating Model Performance
Hands-on Practical: Training a Classifier on MNIST
Quiz for Chapter 5
Chapter 6: Regularization and Improving Performance
The Problem of Overfitting
Regularization Techniques Overview
L1 and L2 Regularization
Dropout Regularization
Early Stopping
Batch Normalization
Hyperparameter Tuning Fundamentals
Strategies for Hyperparameter Search (Grid Search, Random Search)
Hands-on Practical: Applying Dropout and Early Stopping
Quiz for Chapter 6
Chapter 7: Introduction to Specialized Architectures
Limitations of Feedforward Networks
Convolutional Neural Networks (CNNs): Motivation
Core CNN Operations: Convolution
Core CNN Operations: Pooling
Typical CNN Architecture
Recurrent Neural Networks (RNNs): Motivation
The Concept of Recurrence and Hidden State
Basic RNN Architecture
Challenges with Simple RNNs (Vanishing/Exploding Gradients)
Overview: LSTMs and GRUs
Quiz for Chapter 7
Designing and Optimizing a Deep Neural Network from Scratch

Core CNN Operations: Convolution

Standard feedforward networks treat input features independently, losing spatial relationships inherent in data like images. Convolutional Neural Networks address this by using a specialized operation called convolution as their core building block. This operation allows the network to learn and detect local patterns within the input, preserving spatial hierarchies.

The Filter (Kernel): A Feature Detector

At the core of the convolution operation is the filter, also known as a kernel. Think of a filter as a small, learnable matrix of weights. Its purpose is to scan the input data and detect specific features or patterns. For example, in an image, one filter might learn to detect vertical edges, another might detect horizontal edges, and yet another might respond strongly to a particular texture or color gradient.

Filters are typically small spatially (e.g., 3x3, 5x5 pixels) but extend through the full depth of the input volume. If the input is a color image (with Red, Green, Blue channels), a filter will also have a depth of 3.

The Convolution Process: Sliding and Computing

The convolution operation involves sliding the filter across the input data (like an image or the feature map from a previous layer) systematically. At each position, the filter overlays a small patch of the input. The core computation involves two steps:

  1. Element-wise Multiplication: Multiply the weights in the filter by the corresponding values in the input patch it currently covers.
  2. Summation: Sum up all the results from the element-wise multiplication.
  3. Bias Addition (Optional): Often, a single learnable bias term is added to the sum.

This single computed value (sum + bias) represents the response of the filter at that specific location in the input. A strong positive value indicates a strong presence of the feature the filter is designed to detect.

The filter then slides to the next position, and the process repeats until the entire input has been covered.

A 3x3 filter overlaying a region of the input. The corresponding elements are multiplied, and the results are summed (plus bias) to produce one value in the output feature map. The filter then moves to the next position.

Feature Maps (Activation Maps)

The output generated by applying a single filter across the entire input is a 2D matrix called a feature map or activation map. Each element in this map corresponds to the filter's response at a specific spatial location in the input. High activation values signify that the feature detected by the filter is strongly present at that location.

Typically, a convolutional layer uses multiple filters (e.g., 32, 64, or more), each initialized differently and thus learning to detect different features. Applying all these filters to the same input volume results in a stack of 2D feature maps, forming the output volume of the convolutional layer. The depth of this output volume equals the number of filters used.

Controlling the Slide: Stride

The stride defines the step size the filter takes as it moves across the input.

  • A stride of 1 (S=1S=1S=1) means the filter shifts one pixel at a time (horizontally and vertically). This leads to overlapping receptive fields and typically larger output feature maps.
  • A stride of 2 (S=2S=2S=2) means the filter shifts its starting position two pixels at a time (horizontally and vertically). This means it moves from position xxx to x+2x+2x+2. This results in downsampling and less overlap.

Choosing the stride is a design decision that impacts the spatial dimensions of the output and the computational cost.

Preserving Dimensions: Padding

When a filter is applied, especially near the borders of the input, the output feature map naturally tends to shrink in size compared to the input. Furthermore, pixels at the very edge of the input are covered by the filter fewer times than pixels in the center, potentially leading to loss of information.

Padding addresses these issues by adding extra pixels (usually with a value of zero) around the border of the input volume before applying the convolution.

There are two common padding strategies:

  1. Valid Padding: This simply means no padding is added. The output size will be smaller than the input size. The formula for the output dimension (assuming square input W×WW \times WW×W and filter F×FF \times FF×F with stride SSS) is: Wout=⌊W−FS⌋+1W_{out} = \lfloor \frac{W - F}{S} \rfloor + 1Wout​=⌊SW−F​⌋+1
  2. Same Padding: The goal here is to add just enough zero-padding so that the output feature map has the same height and width as the input feature map (when using a stride of S=1S=1S=1). This is very common as it allows building deeper networks without worrying about the spatial dimensions shrinking drastically at each layer. The amount of padding PPP needed for 'same' padding (with S=1S=1S=1) is typically P=(F−1)/2P = (F - 1) / 2P=(F−1)/2 (for odd filter sizes).

Summary of Convolution

In essence, the convolution operation uses learnable filters to scan input data, performing element-wise multiplications and summations to create feature maps. These maps highlight the presence of specific patterns (learned by the filters) at different spatial locations. Parameters like the number of filters, filter size, stride, and padding allow control over how features are extracted and the spatial dimensions of the output. This process forms the foundation for how CNNs effectively analyze grid-like data like images.

Here's a simple example using PyTorch to define a 2D convolutional layer:

import torch
import torch.nn as nn

# Example: Input with 3 channels (e.g., RGB image), batch size 1
# Input dimensions: (batch_size, channels, height, width)
input_tensor = torch.randn(1, 3, 64, 64)

# Define a convolutional layer
# in_channels=3: Matches the input depth (RGB)
# out_channels=16: We want to use 16 different filters
# kernel_size=3: Each filter will be 3x3
# stride=1: Filter moves one pixel at a time
# padding=1: Use 'same' padding (for 3x3 kernel, stride 1)
conv_layer = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1)

# Apply the convolution
output_feature_map = conv_layer(input_tensor)

# Print the output shape
# Output: torch.Size([1, 16, 64, 64])
# Batch size 1, 16 feature maps (one for each filter), height 64, width 64
# Note: Height and width remain 64 due to padding='same' (padding=1) and stride=1.
print(output_feature_map.shape)

This basic operation, repeated across multiple layers, allows CNNs to learn a hierarchy of features, starting from simple edges and textures in early layers to more complex object parts in deeper layers.

  • Gradient-based learning applied to document recognition, Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, 1998 Proceedings of the IEEE, Vol. 86 (IEEE) DOI: 10.1109/5.726791 - This seminal paper introduces LeNet-5, a pioneering convolutional neural network, detailing the convolution operation and its application.
  • Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron Courville, 2016 (MIT Press) - Chapter 9, "Convolutional Networks," provides a comprehensive academic treatment of convolution and related concepts in deep learning.
  • Convolutional Neural Networks (CNNs), Fei-Fei Li, Justin Johnson, Serena Yeung, et al. (Stanford University CS231n Course Notes), 2023 - These widely used course notes offer an intuitive explanation of CNNs, including the convolution operation, filters, stride, and padding.
  • Conv2d, PyTorch Developers, 2024 (PyTorch Foundation) - Official documentation for PyTorch's 2D convolutional layer, providing precise details on its parameters and behavior, complementing the code example.

© 2026 ApX Machine Learning