Basic Tensor Operations

Performing fundamental operations on PyTorch tensors is a primary skill. Tensors support a wide range of mathematical and logical operations, many of which behave similarly to their NumPy counterparts, operating element-wise. These operations form the basis for calculations within neural networks.

Element-wise Arithmetic Operations

The most common operations involve applying standard arithmetic functions independently to each element of the participating tensors. Tensors must typically have compatible shapes for these operations (more on shape compatibility when we discuss broadcasting in the next chapter).

You can use standard Python arithmetic operators or equivalent torch functions:

• Addition: + or torch.add()
• Subtraction: - or torch.sub()
• Multiplication: * or torch.mul()
• Division: / or torch.div()
• Exponentiation: ** or torch.pow()

Let's see these in action:

import torch

# Create two tensors
a = torch.tensor([[1., 2.], [3., 4.]])
b = torch.tensor([[5., 6.], [7., 8.]])

# Addition
sum_tensor = a + b
print("Addition (a + b):\n", sum_tensor)
print("Addition (torch.add(a, b)):\n", torch.add(a, b))

# Subtraction
diff_tensor = a - b
print("\nSubtraction (a - b):\n", diff_tensor)

# Element-wise Multiplication
mul_tensor = a * b
print("\nElement-wise Multiplication (a * b):\n", mul_tensor)
print("Element-wise Multiplication (torch.mul(a, b)):\n", torch.mul(a, b))

# Division
div_tensor = a / b
print("\nDivision (a / b):\n", div_tensor)

# Exponentiation
pow_tensor = a ** 2
print("\nExponentiation (a ** 2):\n", pow_tensor)
print("Exponentiation (torch.pow(a, 2)):\n", torch.pow(a, 2))

These operations create new tensors containing the results. The original tensors a and b remain unchanged.

In-place Operations

PyTorch also offers in-place versions for many operations. These modify the tensor directly without creating a new object, which can save memory. In-place functions are usually identifiable by a trailing underscore _ in their name (e.g., add_, mul_).

import torch

a = torch.tensor([[1., 2.], [3., 4.]])
b = torch.tensor([[5., 6.], [7., 8.]])

print("Original tensor 'a':\n", a)

# Perform in-place addition
a.add_(b) # a is modified directly
print("\nTensor 'a' after a.add_(b):\n", a)

# This would raise an error if uncommented,
# because the result of a + b is a new tensor,
# not suitable for direct assignment back to 'a's memory
# a = a + b # Standard addition creates a new tensor

# Another in-place operation
a.mul_(2) # Multiply 'a' by 2 in-place
print("\nTensor 'a' after a.mul_(2):\n", a)

While in-place operations can be memory efficient, use them with caution. Modifying tensors in-place can cause issues with gradient calculations in Autograd (covered in Chapter 3) if the original value was needed elsewhere in the computation graph. It's often safer, especially when learning, to use standard operations that return new tensors.

Scalar Operations

You can perform arithmetic operations between a tensor and a single number (a scalar). PyTorch automatically expands the scalar to match the tensor's shape for the element-wise operation.

import torch

t = torch.tensor([[1, 2, 3], [4, 5, 6]])
scalar = 10

# Add scalar
print("t + scalar:\n", t + scalar)

# Multiply by scalar
print("\nt * scalar:\n", t * scalar)

# Subtract scalar
print("\nt - scalar:\n", t - scalar)

Other Mathematical Functions

PyTorch provides a rich library of mathematical functions that operate element-wise on tensors, similar to NumPy's universal functions (ufuncs).

import torch

t = torch.tensor([[1., 4.], [9., 16.]])

# Square root
print("Square Root (torch.sqrt(t)):\n", torch.sqrt(t))

# Exponential
print("\nExponential (torch.exp(t)):\n", torch.exp(t)) # e^x

# Natural Logarithm
# Note: Ensure values are positive for log
t_pos = torch.abs(t) + 1e-6 # Add small epsilon for stability if zeros exist
print("\nNatural Log (torch.log(t_pos)):\n", torch.log(t_pos))

# Absolute value
t_neg = torch.tensor([[-1., 2.], [-3., 4.]])
print("\nAbsolute Value (torch.abs(t_neg)):\n", torch.abs(t_neg))

Many other functions like torch.sin(), torch.cos(), torch.tanh(), torch.sigmoid() are available in the torch module.

Reduction Operations

Reduction operations reduce the number of elements in a tensor, often summarizing information. Common examples include sum, mean, minimum, and maximum.

import torch

t = torch.tensor([[1., 2., 3.], [4., 5., 6.]])
print("Original Tensor:\n", t)

# Sum of all elements
total_sum = torch.sum(t)
print("\nSum of all elements (torch.sum(t)):", total_sum)

# Mean of all elements
# Note: Requires float tensor for mean calculation
mean_val = torch.mean(t.float())
print("Mean of all elements (torch.mean(t.float())):", mean_val)

# Max value
max_val = torch.max(t)
print("Max value in tensor (torch.max(t)):", max_val)

# Min value
min_val = torch.min(t)
print("Min value in tensor (torch.min(t)):", min_val)

You can also perform reductions along a specific dimension using the dim argument. This collapses the specified dimension, returning a tensor with one fewer dimension.

import torch

t = torch.tensor([[1., 2., 3.], [4., 5., 6.]])
print("Original Tensor:\n", t)

# Sum along dimension 0 (summing rows)
sum_dim0 = torch.sum(t, dim=0)
print("\nSum along dim=0 (columns):\n", sum_dim0)

# Sum along dimension 1 (summing columns)
sum_dim1 = torch.sum(t, dim=1)
print("\nSum along dim=1 (rows):\n", sum_dim1)

# Mean along dimension 1
mean_dim1 = torch.mean(t.float(), dim=1)
print("\nMean along dim=1 (rows):\n", mean_dim1)

Summing the tensor [[1, 2, 3], [4, 5, 6]] along dim=1 results in [1+2+3, 4+5+6] = [6, 15].

Understanding how dim works is significant for many deep learning operations, like calculating loss per batch item or applying batch normalization.

Comparison Operations

You can compare tensors element-wise using standard comparison operators (>, <, >=, <=, ==, !=). The result is a tensor of boolean values (torch.bool).

import torch

a = torch.tensor([[1, 2], [3, 4]])
b = torch.tensor([[1, 5], [0, 4]])

print("Tensor 'a':\n", a)
print("Tensor 'b':\n", b)

# Equality check
print("\na == b:\n", a == b)

# Greater than check
print("\na > b:\n", a > b)

# Less than or equal check
print("\na <= b:\n", a <= b)

Boolean tensors are useful for masking operations, which you'll see in more advanced tensor manipulations.

Logical Operations

Logical operations (torch.logical_and(), torch.logical_or(), torch.logical_not()) operate element-wise on boolean tensors or tensors that can be evaluated in a boolean context (where 0 is false and non-zero is true).

import torch

bool_a = torch.tensor([[True, False], [True, True]])
bool_b = torch.tensor([[False, True], [True, False]])

print("Boolean Tensor 'bool_a':\n", bool_a)
print("Boolean Tensor 'bool_b':\n", bool_b)

# Logical AND
print("\ntorch.logical_and(bool_a, bool_b):\n", torch.logical_and(bool_a, bool_b))

# Logical OR
print("\ntorch.logical_or(bool_a, bool_b):\n", torch.logical_or(bool_a, bool_b))

# Logical NOT
print("\ntorch.logical_not(bool_a):\n", torch.logical_not(bool_a))

These basic operations provide the building blocks for more complex computations. Mastering them is the first step towards implementing numerical algorithms and neural network layers in PyTorch. In the next chapter, we will look at more advanced ways to manipulate tensors, including indexing, slicing, reshaping, and broadcasting.