Introduction to Tensors

The Tensor is the core data structure for working with PyTorch. If you have experience with NumPy, you will find PyTorch Tensors quite familiar. Tensors are multi-dimensional arrays, very similar to NumPy's ndarray.

Think of tensors as generalizations of familiar mathematical objects:

A scalar (a single number, like 7) is a 0-dimensional tensor (or rank-0 tensor).
A vector (a 1D list or array of numbers, like [1, 2, 3]) is a 1-dimensional tensor (or rank-1 tensor).
A matrix (a 2D grid of numbers, like [[1, 2], [3, 4]]) is a 2-dimensional tensor (or rank-2 tensor).
Extending this, you can have 3-dimensional tensors (like a cube of numbers, often used for RGB images: height x width x channels), 4-dimensional tensors (often used for batches of images: batch_size x height x width x channels), and so on.

A view of tensors as generalizations of scalars, vectors, and matrices, increasing in dimensionality.

In the context of deep learning, tensors are used to represent virtually everything:

Input data: A batch of images, sequences of text, or tables of features.
Model parameters: Weights and biases of neural network layers.
Intermediate activations: The outputs of layers within the network.
Gradients: The values computed during backpropagation used to update model parameters.

What makes PyTorch Tensors particularly suited for deep learning, compared to standard Python lists or even NumPy arrays?

GPU Acceleration: Tensors can be easily moved to and processed on Graphics Processing Units (GPUs) or other hardware accelerators. This provides massive speedups for the computationally intensive operations common in deep learning.
Automatic Differentiation: PyTorch Tensors have built-in support for automatic differentiation through the Autograd system (which we'll cover in Chapter 3). This mechanism automatically calculates gradients, which are fundamental for training neural networks via backpropagation.

While the concept is similar to NumPy arrays, these two features are what make PyTorch Tensors the foundation for building and training models efficiently. In the following sections, we will explore how to create and manipulate these essential data structures.

Was this section helpful?

References

PyTorch Tensors, PyTorch developers, 2024 (PyTorch Foundation) - Official documentation for PyTorch's fundamental data structure, covering its definition, properties, and basic operations.
Automatic differentiation with torch.autograd, PyTorch developers, 2016 (PyTorch Foundation) - Official documentation explaining PyTorch's automatic differentiation engine, which is crucial for neural network training.
Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - A foundational textbook introducing tensors as mathematical objects, including scalars, vectors, and matrices, in the context of deep learning.
Neural Networks Part 1: Setting up the Architecture, Andrej Karpathy, Justin Johnson, Serena Yeung, et al. (Stanford CS231n course staff), 2023 Stanford CS231n Course Notes - Stanford's renowned course notes that offer a practical introduction to tensors as data representations in deep learning, especially for image processing.