ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Linear Algebra Essentials for Machine Learning
Chapter 1: Vectors in Machine Learning
Vectors as Data Representations
Fundamental Vector Operations
Vector Magnitude and Direction
The Dot Product and Projections
Vector Norms: Measuring Length
Calculating Distances Between Vectors
Implementing Vector Operations with NumPy
Hands-on Practical: Feature Vector Manipulation
Quiz for Chapter 1
Chapter 2: Matrices: Data Representation and Transformations
Matrices for Organizing Data
Basic Matrix Operations
Matrix Multiplication Explained
Matrices as Linear Transformations
Common Types of Matrices
Representing Systems of Linear Equations
Implementing Matrix Operations with NumPy
Hands-on Practical: Transforming Data Points
Quiz for Chapter 2
Chapter 3: Solving Linear Systems and Matrix Inverses
Linear Systems in Machine Learning Models
Introduction to Gaussian Elimination
The Matrix Inverse
Calculating the Inverse of a Matrix
The Determinant and Invertibility
Solving Ax=b using the Inverse
Numerical Stability and Alternatives
Hands-on Practical: Solving for Model Coefficients
Quiz for Chapter 3
Chapter 4: Vector Spaces, Subspaces, and Linear Independence
Defining Vector Spaces
Understanding Subspaces
Linear Combinations and Span
Linear Independence of Vectors
Basis and Dimension
Column Space and Null Space
Rank of a Matrix
Hands-on Practical: Analyzing Feature Vector Sets
Quiz for Chapter 4
Chapter 5: Eigenvalues and Eigenvectors
Definition of Eigenvalues and Eigenvectors
Geometric Interpretation
The Characteristic Equation
Calculating Eigenvectors
Eigen-decomposition of a Matrix
Significance in Principal Component Analysis (PCA)
Calculating Eigenvalues/Eigenvectors with NumPy
Hands-on Practical: Eigen-decomposition Calculations
Quiz for Chapter 5
Chapter 6: Matrix Decompositions for Machine Learning
Introduction to Matrix Decompositions
Singular Value Decomposition (SVD)
Geometric Interpretation of SVD
SVD for Dimensionality Reduction
SVD Applications in Data Compression
Relationship between SVD and Eigen-decomposition
Overview of LU Decomposition
Overview of QR Decomposition
Implementing Decompositions with SciPy/NumPy
Hands-on Practical: Applying SVD
Quiz for Chapter 6

SVD Applications in Data Compression

One of the compelling applications of Singular Value Decomposition (SVD) is in data compression, particularly for data that can be naturally represented as matrices, like images. The core idea relies on the property that SVD organizes the information within a matrix in order of importance, allowing us to discard less important information while retaining a good approximation of the original data.

Recall the SVD of an m×nm \times nm×n matrix AAA: A=UΣVTA = U \Sigma V^TA=UΣVT Here, Σ\SigmaΣ is an m×nm \times nm×n diagonal matrix containing the singular values σ1,σ2,σ3,…\sigma_1, \sigma_2, \sigma_3, \dotsσ1​,σ2​,σ3​,… ordered from largest to smallest (σ1≥σ2≥⋯≥0\sigma_1 \ge \sigma_2 \ge \dots \ge 0σ1​≥σ2​≥⋯≥0). These singular values quantify the importance of the corresponding directions captured by the columns of UUU and VVV. Larger singular values correspond to dimensions that capture more of the variance or "energy" of the data in the matrix.

Data compression via SVD works by creating a low-rank approximation of the original matrix AAA. Instead of using all the singular values and vectors, we keep only the first kkk components, where kkk is smaller than the rank of the original matrix. This process is often called truncated SVD.

We construct the approximation AkA_kAk​ as follows: Ak=UkΣkVkTA_k = U_k \Sigma_k V_k^TAk​=Uk​Σk​VkT​ Where:

  • UkU_kUk​ consists of the first kkk columns of UUU (an m×km \times km×k matrix).
  • Σk\Sigma_kΣk​ is a k×kk \times kk×k diagonal matrix containing the first kkk singular values (σ1,…,σk\sigma_1, \dots, \sigma_kσ1​,…,σk​).
  • VkTV_k^TVkT​ consists of the first kkk rows of VTV^TVT (which are the first kkk columns of VVV, transposed, resulting in a k×nk \times nk×n matrix).

The Eckart-Young-Mirsky theorem states that this matrix AkA_kAk​ is the best rank-kkk approximation of AAA in terms of the Frobenius norm (and the spectral norm). Essentially, by keeping the components associated with the largest singular values, we retain the most significant parts of the original matrix AAA.

Why is this Compression?

Storing the original matrix AAA requires storing m×nm \times nm×n numbers. Storing the truncated SVD components UkU_kUk​, Σk\Sigma_kΣk​, and VkTV_k^TVkT​ requires storing:

  • m×km \times km×k values for UkU_kUk​
  • kkk values for the diagonal elements of Σk\Sigma_kΣk​
  • k×nk \times nk×n values for VkTV_k^TVkT​

The total storage is mk+k+kn=k(m+n+1)mk + k + kn = k(m + n + 1)mk+k+kn=k(m+n+1) values. If kkk is significantly smaller than mmm and nnn, then k(m+n+1)k(m + n + 1)k(m+n+1) can be much smaller than m×nm \times nm×n, achieving compression.

Image Compression Example

A grayscale digital image can be represented as a matrix where each entry corresponds to the pixel intensity at a specific location. Let's say we have a 512×512512 \times 512512×512 grayscale image. This matrix AAA has 512×512=262,144512 \times 512 = 262,144512×512=262,144 entries.

  1. Apply SVD: Compute A=UΣVTA = U \Sigma V^TA=UΣVT. UUU and VVV will be 512×512512 \times 512512×512, and Σ\SigmaΣ will be 512×512512 \times 512512×512.
  2. Truncate: Choose a value for kkk. For example, let k=50k=50k=50. Construct A50=U50Σ50V50TA_{50} = U_{50} \Sigma_{50} V_{50}^TA50​=U50​Σ50​V50T​.
  3. Store: Instead of storing the original 262,144 pixel values, we store U50U_{50}U50​ (512×50512 \times 50512×50), Σ50\Sigma_{50}Σ50​ (505050 values), and V50TV_{50}^TV50T​ (50×51250 \times 51250×512). The total storage is 50×512+50+50×512=25600+50+25600=51,25050 \times 512 + 50 + 50 \times 512 = 25600 + 50 + 25600 = 51,25050×512+50+50×512=25600+50+25600=51,250 values. This is roughly 20% of the original storage size.
  4. Reconstruct: To view the compressed image, multiply the stored components: A50=U50Σ50V50TA_{50} = U_{50} \Sigma_{50} V_{50}^TA50​=U50​Σ50​V50T​. The resulting matrix A50A_{50}A50​ is an approximation of the original image AAA.

The choice of kkk determines the trade-off between compression ratio and image quality. A smaller kkk yields higher compression but a potentially lower-quality image reconstruction. A larger kkk results in better fidelity to the original image but less compression. Often, a relatively small kkk can capture most of the visual information, as the singular values tend to decrease rapidly for typical images.

Singular values for an image matrix often show a steep initial drop followed by a long tail of smaller values. This suggests that keeping only the first few dozen or hundred singular values (corresponding to a rank-kkk approximation) can preserve much of the image's structure while discarding components associated with smaller singular values, which might correspond to finer details or noise.

For color images, which typically have separate channels (e.g., Red, Green, Blue), SVD compression can be applied independently to the matrix representing each color channel.

Lossy Compression

It's important to note that SVD-based compression is lossy. The reconstructed matrix AkA_kAk​ is not identical to the original matrix AAA (unless kkk equals the rank of AAA). Some information is permanently discarded during the truncation step. However, because SVD prioritizes the most significant components, the perceptual difference between AAA and AkA_kAk​ can be small, especially for moderate values of kkk, making it effective for applications like image compression where perfect reconstruction is not always necessary.

While image compression is a classic illustration, the principle applies to any data representable as a matrix where a lower-rank approximation is acceptable and beneficial for storage or computational efficiency. This could include large feature matrices in machine learning, user-item interaction matrices in recommendation systems, or numerical simulation data.

  • Numerical Linear Algebra, Lloyd N. Trefethen and David Bau III, 1997 (SIAM) DOI: 10.1137/1.9780898719574 - Covers the theoretical and computational aspects of SVD and its application in low-rank approximations, including the Eckart-Young-Mirsky theorem.
  • Linear Algebra and Learning from Data, Gilbert Strang, 2019 (Wellesley-Cambridge Press) - Discusses SVD as a fundamental tool for data compression and dimensionality reduction in the context of machine learning.
  • Digital Image Processing, Rafael C. Gonzalez and Richard E. Woods, 2016 (Pearson) - Features detailed explanations and examples of SVD for image compression within the field of digital image processing. (4th Edition)
  • 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning, Gilbert Strang, 2018 (MIT OpenCourseWare) - Provides video lectures and course notes that illustrate SVD principles and demonstrate its application to image compression.

© 2026 ApX Machine Learning