Introduction to Vision Transformers

Convolutional Neural Networks (CNNs) excel at capturing local features using their convolutional filters; however, modeling long-range dependencies across an entire image remains a challenge. This is primarily due to the inherently local nature of convolution operations. Significantly increasing receptive fields often necessitates very deep networks or large kernels, which can be computationally expensive and difficult to optimize. While techniques such as attention mechanisms integrated into CNNs offer some benefits, they fundamentally operate within a convolutional framework.

The Transformer architecture, initially developed for natural language processing (NLP), offered a different approach. Transformers rely entirely on self-attention mechanisms, allowing them to model dependencies between any two elements in a sequence, regardless of their distance. This proved highly effective for tasks like machine translation and text generation where understanding the context across long sentences is necessary.

A significant question arose: could this powerful sequence-modeling architecture be adapted for computer vision? Images, unlike text, are not inherently sequential one-dimensional structures. They possess a strong 2D spatial structure, and the number of pixels (potential sequence elements) in a typical image is greatly larger than the number of words in a typical sentence, posing computational challenges for the standard Transformer which has a quadratic complexity with respect to sequence length.

The Vision Transformer (ViT) model represents a successful and influential approach to applying Transformers directly to image classification. Proposed by Dosovitskiy et al. in 2020, the core idea is surprisingly straightforward: treat an image as a sequence of smaller, fixed-size patches.

An input image is divided into a grid of non-overlapping patches. Each patch is then flattened into a vector, forming a sequence that can be processed by a standard Transformer encoder.

Here's how it works at a high level:

1. Image Patching: The input image (e.g., 224x224 pixels) is divided into a grid of fixed-size, non-overlapping patches (e.g., 16x16 pixels).
2. Flattening & Embedding: Each patch is flattened into a single vector (e.g., 16x16x3 = 768 elements for RGB). These vectors are then linearly projected into a desired embedding dimension (e.g., 768 dimensions). This results in a sequence of patch embeddings.
3. Position Embeddings: Since the Transformer architecture itself doesn't inherently understand spatial order, learnable position embeddings are added to the patch embeddings to retain positional information. A special [class] token embedding is often prepended to the sequence, similar to BERT's [CLS] token, whose corresponding output from the Transformer is used for classification.
4. Transformer Encoder: The resulting sequence of vectors (patch embeddings + position embeddings) is fed into a standard Transformer encoder, composed of alternating multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks. The self-attention mechanism allows every patch to attend to every other patch, enabling the model to capture global relationships across the image. Layer normalization is applied before each block, and residual connections are used after each block.
5. Classification Head: Finally, the output corresponding to the [class] token is passed through a small classification head (typically an MLP) to produce the final prediction.

This approach effectively bypasses the need for convolutions, directly applying the sequence-processing power of Transformers to visual data. However, a significant difference compared to CNNs is the lack of strong inductive biases. CNNs have built-in assumptions about locality (pixels nearby are related) and translation equivariance (a feature detected in one location can be detected elsewhere). ViT has much weaker biases, learning relationships almost entirely from data through self-attention. This makes ViT potentially more general but often requires significantly larger datasets for pre-training to achieve performance comparable to or exceeding state-of-the-art CNNs.

The introduction of ViT marked a notable shift in computer vision research, demonstrating that architectures heavily reliant on self-attention could achieve excellent results on image recognition tasks, previously dominated by CNNs. The following sections will provide a more detailed look into the specific architectural components of ViT and compare its characteristics with CNNs.