Representing Systems of Linear Equations

Matrices can organize data, such as holding multiple feature vectors as rows. Matrices also transform vectors when multiplied by them. Combining these two capabilities, matrices provide a powerful and concise way to represent systems of linear equations. This representation is fundamental in many areas, including solving for parameters in machine learning models.

What is a System of Linear Equations?

Recall that a system of linear equations consists of multiple equations that share the same set of variables. For example, finding two unknown values, $x_1$ and $x_2$, that satisfy the following two conditions simultaneously:

$$\begin{cases} 2x_1 + 3x_2 = 7 \\ 1x_1 - 1x_2 = 1 \end{cases}$$

This is a simple system with two equations and two variables. As systems grow larger, with more equations and variables, writing them out this way becomes cumbersome.

From System to Matrix Equation: $Ax = b$

Linear algebra offers a much neater way to express such systems using matrix multiplication. We can separate the system into three distinct components:

1. The Coefficients: The numbers multiplying the variables. We arrange these into a matrix, typically denoted as $A$. Each row in the matrix corresponds to an equation, and each column corresponds to a variable. For our example: $A = \begin{bmatrix} 2 & 3 \\ 1 & -1 \end{bmatrix}$

2. The Variables: The unknown values we want to solve for. We arrange these into a column vector, typically denoted as $x$. For our example: $x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}$

3. The Constants: The values on the right-hand side of the equations. We arrange these into another column vector, typically denoted as $b$. For our example: $b = \begin{bmatrix} 7 \\ 1 \end{bmatrix}$

Now, we can represent the entire system of equations using a single matrix equation:

$Ax = b$

Substituting our matrices and vectors:

$\begin{bmatrix} 2 & 3 \\ 1 & -1 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 7 \\ 1 \end{bmatrix}$

Verifying the Representation

Let's perform the matrix multiplication $Ax$ according to the rules we learned earlier (row-by-column dot products):

The first element of the resulting vector is the dot product of the first row of $A$ and the vector $x$: $(2 \times x_1) + (3 \times x_2)$. The second element is the dot product of the second row of $A$ and the vector $x$: $(1 \times x_1) + (-1 \times x_2)$.

So, the matrix multiplication yields:

$\begin{bmatrix} 2x_1 + 3x_2 \\ 1x_1 - 1x_2 \end{bmatrix}$

Setting this equal to the vector $b$, we get:

$\begin{bmatrix} 2x_1 + 3x_2 \\ 1x_1 - 1x_2 \end{bmatrix} = \begin{bmatrix} 7 \\ 1 \end{bmatrix}$

For two vectors to be equal, their corresponding elements must be equal. This gives us back our original system of equations:

$$\begin{cases} 2x_1 + 3x_2 = 7 \\ 1x_1 - 1x_2 = 1 \end{cases}$$

This confirms that the matrix equation $Ax = b$ is indeed a compact representation of the original system.

Why Use the $Ax = b$ Form?

Representing systems of linear equations in the $Ax = b$ format offers several advantages:

1. Conciseness: It provides a compact mathematical notation that is easier to work with, especially for large systems.
2. Structural Clarity: It clearly separates the known coefficients ($A$), the unknown variables ($x$), and the constants ($b$).
3. Computational Framework: This form is the standard input for highly optimized numerical libraries (like NumPy and SciPy) designed to solve linear systems efficiently. Instead of implementing algorithms like Gaussian elimination manually, you can often pass $A$ and $b$ directly to a library function.
4. Theoretical Leverage: It allows us to apply the full power of linear algebra theory. Concepts like matrix inverses, determinants, and matrix decompositions (which we'll cover later) provide methods and insights for solving these systems and understanding their properties.

In machine learning, systems of linear equations frequently appear when fitting models to data. For instance, finding the optimal weights for a linear regression model often involves solving an equation in the $Ax = b$ form, where $A$ relates to the input features, $x$ represents the model weights we want to find, and $b$ relates to the target values.

This section focused on representing systems of linear equations using matrices. In the next chapter, we will explore methods for actually solving the equation $Ax = b$ to find the unknown vector $x$.