The Matrix Inverse

Many problems in machine learning involve solving systems of linear equations, often represented in the compact form $Ax = b$ where $A$ is a matrix of coefficients (perhaps representing features or model structure), $x$ is a vector of unknowns (like model parameters), and $b$ is a vector of target values or outcomes.

One way to approach solving for $x$ draws inspiration from basic algebra. If you have an equation like $5x = 10$, you solve for $x$ by multiplying both sides by the reciprocal of 5, which is $1/5$ or $5^{-1}$. This gives $(5^{-1})5x = (5^{-1})10$, simplifying to $1x = 2$, or $x=2$. The number $5^{-1}$ is the multiplicative inverse of 5 because $5 \times 5^{-1} = 1$.

Can we do something similar for matrix equations? We need the matrix equivalent of a reciprocal, something that "undoes" the effect of matrix $A$. This is called the matrix inverse.

Definition of the Matrix Inverse

For a given square matrix $A$, its inverse, denoted as $A^{-1}$, is a matrix such that when multiplied by $A$ (in either order), the result is the identity matrix $I$.

$AA^{-1} = A^{-1}A = I$

The identity matrix $I$ (a square matrix with 1s on the main diagonal and 0s elsewhere) acts like the number 1 in matrix multiplication: $AI = IA = A$. So, the definition $AA^{-1} = I$ is directly analogous to $a \times a^{-1} = 1$ in scalar arithmetic.

Important Note: The matrix inverse is defined only for square matrices (matrices with the same number of rows and columns, like $n \times n$). However, not all square matrices have an inverse. Matrices that do have an inverse are called invertible or non-singular. Matrices that do not have an inverse are called non-invertible or singular. We'll see how to determine if a matrix is invertible in the section on determinants.

Let's look at a concrete example. Consider the matrix $A$: $A = \begin{pmatrix} 2 & 1 \\ 1 & 1 \end{pmatrix}$ Its inverse is: $A^{-1} = \begin{pmatrix} 1 & -1 \\ -1 & 2 \end{pmatrix}$ We can verify this by multiplying them: $AA^{-1} = \begin{pmatrix} 2 & 1 \\ 1 & 1 \end{pmatrix} \begin{pmatrix} 1 & -1 \\ -1 & 2 \end{pmatrix} = \begin{pmatrix} (2)(1)+(1)(-1) & (2)(-1)+(1)(2) \\ (1)(1)+(1)(-1) & (1)(-1)+(1)(2) \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} = I$ Multiplying in the other order, $A^{-1}A$, also yields the identity matrix $I$.

Properties of the Inverse

Matrix inverses have several useful properties:

1. Inverse of the Inverse: The inverse of $A^{-1}$ is $A$ itself. $(A^{-1})^{-1} = A$
2. Inverse of a Product: The inverse of a product of two invertible matrices is the product of their inverses in reverse order. $(AB)^{-1} = B^{-1}A^{-1}$ Think about putting on socks ($A$) then shoes ($B$). To reverse the process ($(AB)^{-1}$), you must take off your shoes first ($B^{-1}$) and then your socks ($A^{-1}$).
3. Inverse of a Transpose: The inverse of the transpose of a matrix is the transpose of its inverse. $(A^T)^{-1} = (A^{-1})^T$

These properties are frequently used when manipulating matrix equations in machine learning derivations.

Geometric Interpretation

Thinking back to Chapter 2, where we saw matrices as linear transformations, the inverse matrix $A^{-1}$ represents the transformation that reverses the transformation performed by $A$. If $A$ rotates and scales space in a certain way, applying $A^{-1}$ afterwards will rotate and scale it back to its original state. Applying $A$ and then $A^{-1}$ (or vice-versa) results in the identity transformation (represented by $I$), which leaves everything unchanged. This aligns with the algebraic definition $AA^{-1} = I$.

Why is the Inverse Significant?

The primary reason the matrix inverse is significant in this context is that it gives us a formal way to solve the linear system $Ax = b$. If $A$ is invertible, we can multiply both sides of the equation on the left by $A^{-1}$:

$$\begin{align*} Ax &= b \\ A^{-1}(Ax) &= A^{-1}b && \text{Left-multiply by } A^{-1} \\ (A^{-1}A)x &= A^{-1}b && \text{Associativity of matrix multiplication} \\ Ix &= A^{-1}b && \text{Definition of inverse } (A^{-1}A = I) \\ x &= A^{-1}b && \text{Property of identity matrix } (Ix = x)\end{align*}$$

This elegant result, $x = A^{-1}b$, tells us that if we can find the inverse of the coefficient matrix $A$, we can find the solution vector $x$ by simply multiplying $A^{-1}$ by the vector $b$.

This provides a powerful tool for understanding solutions to linear systems. In the following sections, we will explore how to calculate the inverse and the determinant (which tells us if the inverse exists). We will also discuss why, despite the elegance of the formula $x = A^{-1}b$, directly computing the inverse is often not the most numerically stable or efficient way to solve linear systems in practice, especially for large matrices encountered in machine learning.