Hands-On Practical: Finding Eigenvalues with NumPy

Solving the characteristic equation by hand provides valuable intuition but is impractical for the matrices encountered in machine learning. For computational efficiency, numerical libraries such as NumPy are essential for finding eigenvalues and eigenvectors.

Using numpy.linalg.eig

NumPy's linear algebra module, linalg, contains the function eig() which does exactly what we need. It takes a square matrix as input and returns two NumPy objects:

1. A 1D array containing the eigenvalues.
2. A 2D array where each column is the eigenvector corresponding to the eigenvalue at the same index in the first array.

Let's start with a simple 2x2 matrix. Consider the transformation matrix $A$:

$$A = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}$$

Now, let's use NumPy to find its eigenvalues and eigenvectors.

import numpy as np

# Define our square matrix
A = np.array([[4, -2],
              [1,  1]])

# Compute the eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)

print("Eigenvalues:")
print(eigenvalues)

print("\nEigenvectors (each column is an eigenvector):")
print(eigenvectors)

Running this code produces the following output:

Eigenvalues:
[3. 2.]

Eigenvectors (each column is an eigenvector):
[[0.89442719 0.70710678]
 [0.4472136  0.70710678]]

From this result, we can identify our eigenvalue-eigenvector pairs:

• The first eigenvalue is $\lambda_1 = 3$, with a corresponding eigenvector $v_1 = \begin{pmatrix} 0.894 \\ 0.447 \end{pmatrix}$.
• The second eigenvalue is $\lambda_2 = 2$, with a corresponding eigenvector $v_2 = \begin{pmatrix} 0.707 \\ 0.707 \end{pmatrix}$.

Notice that the eigenvectors returned by NumPy are unit vectors, meaning their length (L2 norm) is 1. This is a standard convention in numerical libraries, as the direction of an eigenvector is what matters, not its magnitude.

Verifying the Eigen-Equation

To solidify our results, we can check if they satisfy the defining equation $Av = \lambda v$. Let's test this for our first eigenvalue, $\lambda_1 = 3$, and its eigenvector, $v_1$.

First, we isolate the eigenvector from our eigenvectors matrix. Remember that it's the first column.

# Isolate the first eigenvalue and eigenvector
lambda1 = eigenvalues[0]
v1 = eigenvectors[:, 0] # First column

print("Lambda 1:", lambda1)
print("Vector 1:", v1)

Now, let's compute both sides of the equation.

# Left side of the equation: A @ v1
left_side = A @ v1

# Right side of the equation: lambda1 * v1
right_side = lambda1 * v1

print("\nA @ v1 =", left_side)
print("lambda1 * v1 =", right_side)

The output will be:

A @ v1 = [2.68328157 1.34164079]
lambda1 * v1 = [2.68328157 1.34164079]

The two results are identical, confirming that we have found a valid eigenvalue and eigenvector pair. You can perform the same check for the second pair ($\lambda_2=2$ and $v_2$) to further validate the results.

A Geometric View of Eigenvectors

We can visualize this relationship to better understand what "unchanged direction" means. The following plot shows the original eigenvector $v_1$ (in blue) and the transformed vector $A v_1$ (in orange).

The original eigenvector $v_1$ and the vector resulting from the transformation $A v_1$. Both vectors lie on the exact same line extending from the origin. The transformation only scaled the vector by a factor of its eigenvalue ($\lambda_1 = 3$).

As the plot shows, applying matrix $A$ to its eigenvector $v_1$ did not rotate it. It simply stretched it, pointing it in the same direction. The length of the new vector is three times the length of the original, which corresponds exactly to the eigenvalue $\lambda_1 = 3$. This is the geometric meaning of the eigen-equation in action.

This ability to compute the primary axes of a linear transformation is a powerful tool. In the final chapter, we will see how this process forms the backbone of Principal Component Analysis (PCA), a widely used technique for dimensionality reduction in machine learning.