Matrices as Linear Transformations

Matrix multiplication, while often understood as a computational procedure, also carries significant geometric meaning. Multiplying a matrix by a vector represents a way to transform that vector (or the point it represents) in space. This concept of transformation is fundamental in machine learning, used for tasks ranging from changing coordinate systems to manipulating data representations.

Think of a matrix $A$ as an operator or function that takes an input vector $x$ and produces an output vector $y$ through multiplication: $y = Ax$. This operation is called a linear transformation.

What makes a transformation "linear"? It must satisfy two properties for any vectors $u, v$ and any scalar $c$:

1. $A(u + v) = Au + Av$ (preserves addition)
2. $A(cu) = c(Au)$ (preserves scalar multiplication)

Matrix multiplication inherently satisfies these conditions. Geometrically, linear transformations preserve the origin (the zero vector maps to the zero vector) and keep grid lines parallel and evenly spaced, although the spacing and orientation might change. They can stretch, shrink, rotate, reflect, or shear space, but they don't curve it.

Let's examine some common linear transformations represented by 2x2 matrices acting on 2D vectors. We'll visualize how they affect the vertices of a unit square: $(0,0), (1,0), (1,1), (0,1)$.

Scaling

A scaling transformation stretches or shrinks space along the coordinate axes. It's represented by a diagonal matrix.

Consider the matrix $S = \begin{bmatrix} 2 & 0 \\ 0 & 0.5 \end{bmatrix}$. Let's apply it to a vector $x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}$:

$Sx = \begin{bmatrix} 2 & 0 \\ 0 & 0.5 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 2x_1 \\ 0.5x_2 \end{bmatrix}$

This transformation doubles the $x_1$ component and halves the $x_2$ component of any vector. Applying this to our unit square vertices:

• $(0,0) \rightarrow (0, 0)$
• $(1,0) \rightarrow (2, 0)$
• $(1,1) \rightarrow (2, 0.5)$
• $(0,1) \rightarrow (0, 0.5)$

Scaling transformation [[2, 0], [0, 0.5]] applied to the unit square. The square is stretched horizontally and compressed vertically.

Rotation

A rotation transformation rotates vectors around the origin by a specific angle $\theta$. The standard 2D rotation matrix is:

$R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$

Let's rotate by $\theta = 90^\circ$ ($\cos 90^\circ = 0, \sin 90^\circ = 1$). The matrix becomes $R_{90} = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}$.

Applying this to our unit square vertices:

• $(0,0) \rightarrow (0, 0)$
• $(1,0) \rightarrow (0, 1)$
• $(1,1) \rightarrow (-1, 1)$
• $(0,1) \rightarrow (-1, 0)$

Rotation transformation [[0, -1], [1, 0]] applied to the unit square, rotating it 90 degrees counter-clockwise.

Shearing

A shear transformation slants the space, effectively sliding layers relative to each other. A horizontal shear is often represented by a matrix like $H = \begin{bmatrix} 1 & s \\ 0 & 1 \end{bmatrix}$, where $s$ is the shear factor.

Let's use $s = 1$, so $H = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}$. Applying this to a vector $x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}$:

$Hx = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} x_1 + x_2 \\ x_2 \end{bmatrix}$

The transformation adds the $x_2$ coordinate to the $x_1$ coordinate, shifting points horizontally based on their height. Applying this to our unit square vertices:

• $(0,0) \rightarrow (0, 0)$
• $(1,0) \rightarrow (1, 0)$
• $(1,1) \rightarrow (1+1, 1) = (2, 1)$
• $(0,1) \rightarrow (0+1, 1) = (1, 1)$

Horizontal shear transformation [[1, 1], [0, 1]] applied to the unit square. The top edge is shifted right.

Composing Transformations

What happens if we apply one transformation followed by another? For example, first rotate by 90 degrees ($R_{90}$) and then scale ($S$)? We apply the transformations sequentially:

1. $x' = R_{90} x$
2. $y = S x' = S (R_{90} x)$

Because matrix multiplication is associative, this is equivalent to $y = (S R_{90}) x$. The combined transformation is represented by the product of the individual transformation matrices.

Important: Matrix multiplication is generally not commutative ($AB \neq BA$). This means the order in which you apply transformations matters. Scaling then rotating typically yields a different result than rotating then scaling. The matrix corresponding to the transformation applied first appears on the right side of the product.

Relevance to Machine Learning

Understanding matrices as linear transformations is significant in ML:

• Feature Space Transformation: Techniques like Principal Component Analysis (PCA) find new coordinate systems (bases) for your data by applying rotations and scaling (represented by matrices) to maximize variance along axes. Eigen-decomposition and SVD, covered later, are built on this.
• Data Augmentation: In computer vision, rotating, scaling, or shearing images (represented as matrices or tensors of pixel values) creates variations of existing data to improve model generalization.
• Neural Network Layers: Some layers in neural networks perform affine transformations ($Ax + b$), where the matrix multiplication $Ax$ is a linear transformation component.

In the next sections and chapters, we'll see how these geometric interpretations connect to solving linear systems, understanding vector spaces, and performing powerful matrix decompositions used throughout machine learning. Remember that NumPy provides efficient tools (like the @ operator or np.dot()) to perform these matrix multiplications, effectively applying the transformations we've discussed.