Introduction to Tensors

The Tensor is the core data structure for working with PyTorch. If you have experience with NumPy, you will find PyTorch Tensors quite familiar. Tensors are multi-dimensional arrays, very similar to NumPy's ndarray.

Think of tensors as generalizations of familiar mathematical objects:

• A scalar (a single number, like 7) is a 0-dimensional tensor (or rank-0 tensor).
• A vector (a 1D list or array of numbers, like [1, 2, 3]) is a 1-dimensional tensor (or rank-1 tensor).
• A matrix (a 2D grid of numbers, like [[1, 2], [3, 4]]) is a 2-dimensional tensor (or rank-2 tensor).
• Extending this, you can have 3-dimensional tensors (like a cube of numbers, often used for RGB images: height x width x channels), 4-dimensional tensors (often used for batches of images: batch_size x height x width x channels), and so on.

A view of tensors as generalizations of scalars, vectors, and matrices, increasing in dimensionality.

In the context of deep learning, tensors are used to represent virtually everything:

• Input data: A batch of images, sequences of text, or tables of features.
• Model parameters: Weights and biases of neural network layers.
• Intermediate activations: The outputs of layers within the network.
• Gradients: The values computed during backpropagation used to update model parameters.

What makes PyTorch Tensors particularly suited for deep learning, compared to standard Python lists or even NumPy arrays?

1. GPU Acceleration: Tensors can be easily moved to and processed on Graphics Processing Units (GPUs) or other hardware accelerators. This provides massive speedups for the computationally intensive operations common in deep learning.
2. Automatic Differentiation: PyTorch Tensors have built-in support for automatic differentiation through the Autograd system (which we'll cover in Chapter 3). This mechanism automatically calculates gradients, which are fundamental for training neural networks via backpropagation.

While the concept is similar to NumPy arrays, these two features are what make PyTorch Tensors the foundation for building and training models efficiently. In the following sections, we will explore how to create and manipulate these essential data structures.