ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Linear Algebra Essentials for Machine Learning
Chapter 1: Vectors in Machine Learning
Vectors as Data Representations
Fundamental Vector Operations
Vector Magnitude and Direction
The Dot Product and Projections
Vector Norms: Measuring Length
Calculating Distances Between Vectors
Implementing Vector Operations with NumPy
Hands-on Practical: Feature Vector Manipulation
Quiz for Chapter 1
Chapter 2: Matrices: Data Representation and Transformations
Matrices for Organizing Data
Basic Matrix Operations
Matrix Multiplication Explained
Matrices as Linear Transformations
Common Types of Matrices
Representing Systems of Linear Equations
Implementing Matrix Operations with NumPy
Hands-on Practical: Transforming Data Points
Quiz for Chapter 2
Chapter 3: Solving Linear Systems and Matrix Inverses
Linear Systems in Machine Learning Models
Introduction to Gaussian Elimination
The Matrix Inverse
Calculating the Inverse of a Matrix
The Determinant and Invertibility
Solving Ax=b using the Inverse
Numerical Stability and Alternatives
Hands-on Practical: Solving for Model Coefficients
Quiz for Chapter 3
Chapter 4: Vector Spaces, Subspaces, and Linear Independence
Defining Vector Spaces
Understanding Subspaces
Linear Combinations and Span
Linear Independence of Vectors
Basis and Dimension
Column Space and Null Space
Rank of a Matrix
Hands-on Practical: Analyzing Feature Vector Sets
Quiz for Chapter 4
Chapter 5: Eigenvalues and Eigenvectors
Definition of Eigenvalues and Eigenvectors
Geometric Interpretation
The Characteristic Equation
Calculating Eigenvectors
Eigen-decomposition of a Matrix
Significance in Principal Component Analysis (PCA)
Calculating Eigenvalues/Eigenvectors with NumPy
Hands-on Practical: Eigen-decomposition Calculations
Quiz for Chapter 5
Chapter 6: Matrix Decompositions for Machine Learning
Introduction to Matrix Decompositions
Singular Value Decomposition (SVD)
Geometric Interpretation of SVD
SVD for Dimensionality Reduction
SVD Applications in Data Compression
Relationship between SVD and Eigen-decomposition
Overview of LU Decomposition
Overview of QR Decomposition
Implementing Decompositions with SciPy/NumPy
Hands-on Practical: Applying SVD
Quiz for Chapter 6

Significance in Principal Component Analysis (PCA)

Eigenvalues and eigenvectors reveal directions in space that remain unchanged (except for scaling) when a linear transformation is applied. This property is remarkably useful for understanding the underlying structure within data, forming the foundation of Principal Component Analysis (PCA).

PCA is a foundational technique in machine learning and data analysis, primarily used for dimensionality reduction. The main objective is to identify a new set of coordinate axes, called principal components, for the data. These new axes are chosen such that the maximum possible variance in the data is captured along the first axis, the maximum remaining variance along the second axis (orthogonal to the first), and so on. This allows us to represent the data using fewer dimensions while minimizing information loss.

The Covariance Matrix: Capturing Data Spread

To find these directions of maximum variance, we first need to quantify how the different features in our dataset vary together. This is precisely what the covariance matrix (CCC) does. For a dataset with ppp features, the covariance matrix is a p×pp \times pp×p symmetric matrix where:

  • The diagonal element CiiC_{ii}Cii​ is the variance of the iii-th feature.
  • The off-diagonal element CijC_{ij}Cij​ is the covariance between the iii-th feature and the jjj-th feature.

This matrix summarizes the spread and linear relationships within the data.

Eigenvectors as Directions of Maximum Variance

Here's the critical connection: The eigenvectors of the data's covariance matrix CCC point along the directions of maximum variance in the data. These directions are the principal components. Furthermore, the eigenvalue λ\lambdaλ associated with each eigenvector quantifies the amount of variance along that specific direction.

Think back to the definition Ax=λxAx = \lambda xAx=λx. In the context of PCA, our matrix AAA is the covariance matrix CCC. The eigenvectors xxx (often denoted as uuu or vvv in PCA literature) are the principal components. When we apply the transformation represented by CCC to one of its eigenvectors uuu, the result is simply the same vector scaled by its eigenvalue λ\lambdaλ: Cu=λuCu = \lambda uCu=λu.

Why is this important? It turns out that finding the direction uuu (a unit vector) that maximizes the variance of the data projected onto it (uTCuu^T C uuTCu) leads directly to the eigenvector equation. The direction uuu that maximizes this variance is the eigenvector corresponding to the largest eigenvalue of CCC. The maximum variance achieved is equal to that largest eigenvalue λmax\lambda_{max}λmax​.

Principal Components: The New Axes

So, the principal components of the data are simply the eigenvectors of its covariance matrix CCC, typically ordered from highest eigenvalue to lowest.

  1. First Principal Component (PC1): The eigenvector associated with the largest eigenvalue (λ1\lambda_1λ1​). This direction captures the most variance in the dataset.
  2. Second Principal Component (PC2): The eigenvector associated with the second-largest eigenvalue (λ2\lambda_2λ2​). This direction is orthogonal (perpendicular) to PC1 and captures the maximum remaining variance.
  3. Subsequent Components: Each subsequent principal component is orthogonal to all preceding components and captures the maximum variance possible from what's left.

Visualizing Principal Components

Imagine a scatter plot of 2D data points that form an elliptical cloud. The eigenvectors of the covariance matrix of this data will align with the axes of this ellipse.

Scatter plot showing 2D data points. PC1 (red line) indicates the direction of maximum variance, determined by the eigenvector with the largest eigenvalue. PC2 (orange dotted line), orthogonal to PC1, captures the next largest variance.

How PCA Reduces Dimensions

The eigenvalues tell us the amount of variance captured by each principal component (eigenvector). By ranking the eigenvectors according to their corresponding eigenvalues in descending order, we know which directions are most "important" for describing the data's spread.

To reduce data from ppp dimensions to a smaller number kkk (where k<pk < pk<p), PCA involves these steps:

  1. Calculate the covariance matrix CCC of the original data.
  2. Compute the eigenvalues and eigenvectors of CCC.
  3. Select the kkk eigenvectors that correspond to the kkk largest eigenvalues. These form the new basis for our reduced-dimension subspace.
  4. Transform the original data points into this new subspace by projecting them onto the selected kkk eigenvectors.

The result is a kkk-dimensional dataset that retains the most significant variance present in the original ppp-dimensional data. The proportion of total variance retained by the first kkk components can be calculated by summing the top kkk eigenvalues and dividing by the sum of all eigenvalues.

Summary

Eigenvalues and eigenvectors are fundamental to Principal Component Analysis because:

  • Eigenvectors of the covariance matrix define the principal components: the orthogonal axes along which the data exhibits maximum variance.
  • Eigenvalues measure the magnitude of variance along these principal component directions.

This relationship allows PCA to systematically identify the most informative directions in the data, making it possible to reduce dimensionality effectively while preserving essential data structure. Understanding this connection is significant for anyone applying or interpreting PCA results in machine learning. You will learn how to perform these calculations using libraries like NumPy in the upcoming practical exercises.

  • On Lines and Planes of Closest Fit to Systems of Points in Space, Karl Pearson, 1901 Philosophical Magazine, Series 6, Vol. 2 (Taylor & Francis) DOI: 10.1080/14786440109462720 - A foundational paper that geometrically introduced the concept of principal axes, which is the precursor to Principal Component Analysis.
  • The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Trevor Hastie, Robert Tibshirani, and Jerome Friedman, 2009 (Springer) - A widely respected textbook in statistical learning, providing a thorough treatment of PCA as a dimensionality reduction technique within a broader machine learning context.
  • Linear Algebra and Learning from Data, Gilbert Strang, 2019 (Wellesley-Cambridge Press) - A textbook directly connecting linear algebra concepts like eigenvalues and eigenvectors to their use in machine learning algorithms, including PCA.
  • Principal Component Analysis (PCA), Andrew Ng, 2020 Stanford CS229 Machine Learning Course Notes (Stanford University) - Course notes from a highly influential machine learning curriculum, presenting a practical and intuitive explanation of PCA's mathematical foundations and applications.

© 2026 ApX Machine Learning