Basic Matrix Operations

Matrices are structures for organizing data, and fundamental operations can be performed on them. Like operations on vectors, these operations have both mathematical definitions and practical applications in data manipulation and algorithm implementation. We will examine addition, subtraction, scalar multiplication, and transposition. Performing these operations efficiently is important, and NumPy makes this straightforward.

Matrix Addition and Subtraction

Adding or subtracting matrices is intuitive if you're familiar with vector addition. The rule is simple: you can only add or subtract matrices if they have the exact same dimensions (same number of rows and same number of columns). The operation is performed element-wise.

If you have two matrices, $A$ and $B$, both with dimensions $m \times n$, their sum $C = A + B$ is also an $m \times n$ matrix where each element $c_{ij}$ is the sum of the corresponding elements from $A$ and $B$:

$$c_{ij} = a_{ij} + b_{ij}$$

Similarly, for subtraction $D = A - B$, each element $d_{ij}$ is:

$$d_{ij} = a_{ij} - b_{ij}$$

Example:

Let's say we have two $2 \times 3$ matrices:

$$A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}, \quad B = \begin{bmatrix} 7 & 8 & 9 \\ 0 & 1 & 2 \end{bmatrix}$$

Their sum $A + B$ is:

$$A + B = \begin{bmatrix} 1+7 & 2+8 & 3+9 \\ 4+0 & 5+1 & 6+2 \end{bmatrix} = \begin{bmatrix} 8 & 10 & 12 \\ 4 & 6 & 8 \end{bmatrix}$$

Their difference $A - B$ is:

$$A - B = \begin{bmatrix} 1-7 & 2-8 & 3-9 \\ 4-0 & 5-1 & 6-2 \end{bmatrix} = \begin{bmatrix} -6 & -6 & -6 \\ 4 & 4 & 4 \end{bmatrix}$$

Implementation with NumPy:

NumPy makes these operations trivial using the standard + and - operators.

import numpy as np

# Define matrices A and B
A = np.array([[1, 2, 3],
              [4, 5, 6]])

B = np.array([[7, 8, 9],
              [0, 1, 2]])

# Matrix Addition
C = A + B
print("Matrix A:\n", A)
print("\nMatrix B:\n", B)
print("\nMatrix Sum (A + B):\n", C)

# Matrix Subtraction
D = A - B
print("\nMatrix Difference (A - B):\n", D)

# Attempting addition with different shapes will raise an error
# E = np.array([[1, 2], [3, 4]])
# try:
#     F = A + E
# except ValueError as e:
#     print(f"\nError adding A and E: {e}")

Relevance in ML: While less common than matrix multiplication, addition and subtraction appear when combining or comparing datasets represented as matrices, calculating differences between predicted and actual values (error matrices), or updating weight matrices in certain optimization algorithms.

Scalar Multiplication

Multiplying a matrix by a scalar (a single number) is also an element-wise operation. You simply multiply every element in the matrix by that scalar.

If $A$ is an $m \times n$ matrix and $c$ is a scalar, then the product $cA$ is an $m \times n$ matrix where each element is $c \cdot a_{ij}$.

Example:

Using matrix $A$ from before and a scalar $c=3$:

$$A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}$$ $$3A = \begin{bmatrix} 3 \times 1 & 3 \times 2 & 3 \times 3 \\ 3 \times 4 & 3 \times 5 & 3 \times 6 \end{bmatrix} = \begin{bmatrix} 3 & 6 & 9 \\ 12 & 15 & 18 \end{bmatrix}$$

Implementation with NumPy:

Use the standard * operator between a scalar and a NumPy array.

import numpy as np

A = np.array([[1, 2, 3],
              [4, 5, 6]])
scalar = 3

# Scalar Multiplication
scaled_A = scalar * A
print("Matrix A:\n", A)
print(f"\nScalar: {scalar}")
print("\nScaled Matrix (scalar * A):\n", scaled_A)

# Can also multiply array by scalar
scaled_A_alt = A * scalar
print("\nScaled Matrix (A * scalar):\n", scaled_A_alt)

Relevance in ML: Scalar multiplication is frequently used when scaling features in a dataset, adjusting learning rates in gradient descent (where you multiply the gradient matrix by the learning rate), or applying regularization terms.

Matrix Transposition

The transpose of a matrix is obtained by swapping its rows and columns. The element at row $i$, column $j$ in the original matrix $A$ moves to row $j$, column $i$ in the transposed matrix, denoted as $A^T$.

If $A$ is an $m \times n$ matrix, its transpose $A^T$ will be an $n \times m$ matrix.

Example:

$$A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix} \quad (2 \times 3 \text{ matrix})$$

Its transpose $A^T$ is:

$$A^T = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix} \quad (3 \times 2 \text{ matrix})$$

Notice how the first row $[1, 2, 3]$ became the first column, and the second row $[4, 5, 6]$ became the second column.

A property to remember is that the transpose of a transpose returns the original matrix: $(A^T)^T = A$.

Implementation with NumPy:

NumPy arrays have a convenient .T attribute for transposition. Alternatively, you can use the np.transpose() function.

import numpy as np

A = np.array([[1, 2, 3],
              [4, 5, 6]])

# Transpose using .T attribute
A_transpose_T = A.T
print("Original Matrix A (2x3):\n", A)
print("\nTranspose using .T (3x2):\n", A_transpose_T)

# Transpose using np.transpose()
A_transpose_func = np.transpose(A)
print("\nTranspose using np.transpose() (3x2):\n", A_transpose_func)

# Transposing a row vector (1D array treated as row)
row_vec = np.array([7, 8, 9]) # Shape (3,)
print("\nOriginal row vector (shape {}):\n".format(row_vec.shape), row_vec)
# Note: .T on a 1D array does nothing!
print("\n.T on 1D array (shape {}):\n".format(row_vec.T.shape), row_vec.T)

# To transpose a 1D array, first make it 2D (e.g., 1xN or Nx1)
row_vec_2d = row_vec[np.newaxis, :] # Shape (1, 3)
col_vec = row_vec_2d.T            # Shape (3, 1)
print("\nRow vector reshaped to 2D (shape {}):\n".format(row_vec_2d.shape), row_vec_2d)
print("\nTransposed to column vector (shape {}):\n".format(col_vec.shape), col_vec)

Note: Transposing a 1-dimensional NumPy array (like np.array([1, 2, 3])) using .T doesn't change its shape. To treat it as a mathematical row vector and transpose it into a column vector, you often need to explicitly give it two dimensions (e.g., shape (1, n) for a row vector or (n, 1) for a column vector) using techniques like reshape() or adding a new axis with np.newaxis or np.expand_dims().

Relevance in ML: Transposition is a fundamental operation used extensively in linear algebra formulas that appear in machine learning. For example, the normal equation for solving linear regression involves $(X^T X)^{-1} X^T y$. It's also used when calculating covariance matrices ($X^T X$, assuming $X$ is centered) or when manipulating data shapes to match requirements for matrix multiplication or other library functions.

These basic operations form the building blocks for more complex matrix manipulations we'll encounter later, particularly matrix multiplication, which is central to linear transformations. Mastering these fundamentals with NumPy provides a solid base for implementing ML algorithms.