Calculating Eigenvectors

The characteristic equation, $det(A - \lambda I) = 0$, is used to find the eigenvalues ($\lambda$) for a given square matrix $A$. Eigenvalues represent the factors by which eigenvectors are scaled when transformed by the matrix $A$. The goal is to find the actual eigenvectors ($x$) corresponding to each of these eigenvalues.

We start with the fundamental definition relating a matrix, its eigenvalue, and its corresponding eigenvector: $Ax = \lambda x$

Our goal is to find the non-zero vector $x$ that satisfies this equation for a specific eigenvalue $\lambda$. We can rearrange this equation to bring all terms to one side: $Ax - \lambda x = 0$

To proceed, we need to factor out the vector $x$. We can introduce the identity matrix $I$ (of the same dimension as $A$) without changing the equation, as $Ix = x$: $Ax - \lambda I x = 0$

Now, we can factor out $x$: $(A - \lambda I)x = 0$

This equation is a homogeneous system of linear equations. It looks very similar to the form $Mx = 0$, where $M = (A - \lambda I)$. We are looking for the non-zero solutions $x$ to this system.

Why are we guaranteed to find non-zero solutions? Recall that we found the eigenvalues $\lambda$ by solving $det(A - \lambda I) = 0$. A matrix with a determinant of zero is called a singular matrix. Singular matrices are not invertible, and an important property of the system $Mx = 0$ when $M$ is singular is that it has infinitely many non-trivial (non-zero) solutions. These non-zero solutions for $x$ are precisely the eigenvectors associated with the eigenvalue $\lambda$.

The Calculation Process

So, for each eigenvalue $\lambda$ you calculated:

1. Form the Matrix: Construct the matrix $M = A - \lambda I$. This involves subtracting the eigenvalue $\lambda$ from each diagonal element of the original matrix $A$.
2. Set up the System: Write down the homogeneous system of equations $Mx = 0$, which is $(A - \lambda I)x = 0$.
3. Solve the System: Find the non-zero solutions for the vector $x$. The standard technique for this is Gaussian elimination (row reduction) applied to the matrix $M = A - \lambda I$. You are essentially finding the null space of the matrix $(A - \lambda I)$.
4. Identify the Eigenvector(s): The solution will typically involve one or more free variables. This means there isn't a single unique solution vector $x$. Instead, the solutions form a subspace called the eigenspace for that eigenvalue $\lambda$. Any non-zero vector in this eigenspace is a valid eigenvector. Usually, we pick a simple representative vector (often by setting the free variable(s) to a convenient value like 1).

Example Calculation

Let's work through a simple 2x2 example. Consider the matrix: $A = \begin{bmatrix} 4 & 1 \\ 2 & 3 \end{bmatrix}$

Suppose we have already found the eigenvalues using the characteristic equation. Let's say one of the eigenvalues is $\lambda = 5$. Now we find the eigenvector(s) corresponding to $\lambda = 5$.

1. Form the Matrix $(A - \lambda I)$: $A - \lambda I = A - 5I = \begin{bmatrix} 4 & 1 \\ 2 & 3 \end{bmatrix} - 5 \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 4 & 1 \\ 2 & 3 \end{bmatrix} - \begin{bmatrix} 5 & 0 \\ 0 & 5 \end{bmatrix} = \begin{bmatrix} -1 & 1 \\ 2 & -2 \end{bmatrix}$

2. Set up the System $(A - \lambda I)x = 0$: Let $x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}$. The system is: $\begin{bmatrix} -1 & 1 \\ 2 & -2 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}$ This translates to the equations: $-x_1 + x_2 = 0$ $2x_1 - 2x_2 = 0$

3. Solve the System: Notice that the second equation is just -2 times the first equation. They are linearly dependent, which is expected since the matrix $(A - 5I)$ is singular. From the first equation, we get: $x_2 = x_1$ The variable $x_1$ is a free variable. We can choose any value for it (except 0, because eigenvectors must be non-zero), and $x_2$ will be determined.

4. Identify an Eigenvector: Let's choose a simple value for the free variable, say $x_1 = 1$. Then $x_2 = 1$. So, one eigenvector corresponding to the eigenvalue $\lambda = 5$ is: $x = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$ Any non-zero scalar multiple of this vector, like $\begin{bmatrix} 2 \\ 2 \end{bmatrix}$ or $\begin{bmatrix} -0.5 \\ -0.5 \end{bmatrix}$, is also a valid eigenvector for $\lambda = 5$. The set of all such vectors forms the eigenspace for $\lambda=5$.

You would repeat this process for any other eigenvalues found for matrix $A$.

While performing Gaussian elimination by hand is instructive for smaller matrices, for larger matrices encountered in machine learning applications, you'll rely on numerical libraries like NumPy, which have efficient functions to compute both eigenvalues and eigenvectors directly. Understanding this underlying process, however, provides valuable insight into what these functions are doing and how to interpret their results.