ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Linear Algebra Essentials for Machine Learning
Chapter 1: Vectors in Machine Learning
Vectors as Data Representations
Fundamental Vector Operations
Vector Magnitude and Direction
The Dot Product and Projections
Vector Norms: Measuring Length
Calculating Distances Between Vectors
Implementing Vector Operations with NumPy
Hands-on Practical: Feature Vector Manipulation
Quiz for Chapter 1
Chapter 2: Matrices: Data Representation and Transformations
Matrices for Organizing Data
Basic Matrix Operations
Matrix Multiplication Explained
Matrices as Linear Transformations
Common Types of Matrices
Representing Systems of Linear Equations
Implementing Matrix Operations with NumPy
Hands-on Practical: Transforming Data Points
Quiz for Chapter 2
Chapter 3: Solving Linear Systems and Matrix Inverses
Linear Systems in Machine Learning Models
Introduction to Gaussian Elimination
The Matrix Inverse
Calculating the Inverse of a Matrix
The Determinant and Invertibility
Solving Ax=b using the Inverse
Numerical Stability and Alternatives
Hands-on Practical: Solving for Model Coefficients
Quiz for Chapter 3
Chapter 4: Vector Spaces, Subspaces, and Linear Independence
Defining Vector Spaces
Understanding Subspaces
Linear Combinations and Span
Linear Independence of Vectors
Basis and Dimension
Column Space and Null Space
Rank of a Matrix
Hands-on Practical: Analyzing Feature Vector Sets
Quiz for Chapter 4
Chapter 5: Eigenvalues and Eigenvectors
Definition of Eigenvalues and Eigenvectors
Geometric Interpretation
The Characteristic Equation
Calculating Eigenvectors
Eigen-decomposition of a Matrix
Significance in Principal Component Analysis (PCA)
Calculating Eigenvalues/Eigenvectors with NumPy
Hands-on Practical: Eigen-decomposition Calculations
Quiz for Chapter 5
Chapter 6: Matrix Decompositions for Machine Learning
Introduction to Matrix Decompositions
Singular Value Decomposition (SVD)
Geometric Interpretation of SVD
SVD for Dimensionality Reduction
SVD Applications in Data Compression
Relationship between SVD and Eigen-decomposition
Overview of LU Decomposition
Overview of QR Decomposition
Implementing Decompositions with SciPy/NumPy
Hands-on Practical: Applying SVD
Quiz for Chapter 6

Linear Combinations and Span

Vectors represent data points, and matrices transform them. Combining vectors to create new ones is a process fundamental to understanding vector spaces. This process is called forming linear combinations.

A linear combination of a set of vectors v1,v2,...,vkv_1, v_2, ..., v_kv1​,v2​,...,vk​ is any vector www that can be expressed in the form:

w=c1v1+c2v2+...+ckvkw = c_1 v_1 + c_2 v_2 + ... + c_k v_kw=c1​v1​+c2​v2​+...+ck​vk​

where c1,c2,...,ckc_1, c_2, ..., c_kc1​,c2​,...,ck​ are scalar constants (real numbers). Essentially, you take your set of vectors, scale each one by some amount (including potentially zero or negative amounts), and then add the scaled vectors together. The result is a linear combination.

Think about it geometrically in 2D. If you have a single vector v1v_1v1​, its linear combinations are just scalar multiples of v1v_1v1​, forming a line passing through the origin in the direction of v1v_1v1​.

If you have two vectors, say v1=[10]v_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}v1​=[10​] and v2=[01]v_2 = \begin{bmatrix} 0 \\ 1 \end{bmatrix}v2​=[01​] in R2\mathbb{R}^2R2, what vectors can you create? A linear combination looks like w=c1[10]+c2[01]=[c1c2]w = c_1 \begin{bmatrix} 1 \\ 0 \end{bmatrix} + c_2 \begin{bmatrix} 0 \\ 1 \end{bmatrix} = \begin{bmatrix} c_1 \\ c_2 \end{bmatrix}w=c1​[10​]+c2​[01​]=[c1​c2​​]. By choosing different scalars c1c_1c1​ and c2c_2c2​, you can reach any point (vector) in the 2D plane.

The Span of Vectors

This leads us to the concept of span. The span of a set of vectors {v1,v2,...,vk}\{v_1, v_2, ..., v_k\}{v1​,v2​,...,vk​} is the set of all possible linear combinations of these vectors. It represents the entire region or space that can be "reached" or generated by combining these vectors.

  • Span of a single non-zero vector: A line through the origin.
  • Span of two non-collinear vectors in R2\mathbb{R}^2R2: The entire 2D plane (R2\mathbb{R}^2R2).
  • Span of two collinear vectors in R2\mathbb{R}^2R2: A line through the origin (the same line defined by either vector alone).
  • Span of two non-collinear vectors in R3\mathbb{R}^3R3: A plane through the origin within the 3D space.
  • Span of three non-coplanar vectors in R3\mathbb{R}^3R3: The entire 3D space (R3\mathbb{R}^3R3).

Let's visualize the span of two non-collinear vectors in 2D. Take v1=[21]v_1 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}v1​=[21​] and v2=[−13]v_2 = \begin{bmatrix} -1 \\ 3 \end{bmatrix}v2​=[−13​]. Their span is the entire R2\mathbb{R}^2R2. Any vector in R2\mathbb{R}^2R2, like w=[45]w = \begin{bmatrix} 4 \\ 5 \end{bmatrix}w=[45​], can be written as a linear combination c1v1+c2v2c_1 v_1 + c_2 v_2c1​v1​+c2​v2​.

Two non-collinear vectors v1v_1v1​ and v2v_2v2​. Any vector www in the plane (like the dotted green vector) can be formed by scaling and adding v1v_1v1​ and v2v_2v2​. The dashed lines represent the infinite lines passing through the origin along v1v_1v1​ and v2v_2v2​. The span fills the entire 2D plane.

Why Span Matters in Machine Learning

In machine learning, features often form vectors. A dataset can be seen as a collection of these feature vectors. The concept of span helps us understand the "reach" of our features.

  • Feature Space: The span of your feature vectors defines the feature space your model effectively operates within. If a new data point lies outside the span of the training data features, the model might struggle to generalize.
  • Redundancy: If one feature vector lies within the span of others, it might be redundant. It doesn't add new "direction" or information to the space defined by the other vectors. This relates closely to the idea of linear independence, which we'll discuss next.
  • Dimensionality: The number of vectors needed to span a particular space relates to its dimension. If your data lives in R100\mathbb{R}^{100}R100 but the feature vectors only span a 10-dimensional subspace, it suggests potential for dimensionality reduction.

Calculating Linear Combinations with NumPy

NumPy makes calculating linear combinations straightforward.

import numpy as np

# Define vectors
v1 = np.array([2, 1])
v2 = np.array([-1, 3])

# Define scalars
c1 = 2.1538 # Approximately 28/13
c2 = 0.3077 # Approximately 4/13

# Calculate the linear combination w = c1*v1 + c2*v2
w = c1 * v1 + c2 * v2

print(f"Vector v1: {v1}")
print(f"Vector v2: {v2}")
print(f"Scalars c1={c1:.4f}, c2={c2:.4f}")
print(f"Linear Combination w: {w}")
# The result should be close to [4, 5] due to floating point representation
# Let's check if w is indeed [4, 5]
target_w = np.array([4, 5])
print(f"Is w approximately equal to [4, 5]? {np.allclose(w, target_w)}")

This code snippet demonstrates creating the vector w=[45]w = \begin{bmatrix} 4 \\ 5 \end{bmatrix}w=[45​] as a linear combination of v1=[21]v_1 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}v1​=[21​] and v2=[−13]v_2 = \begin{bmatrix} -1 \\ 3 \end{bmatrix}v2​=[−13​]. We found the specific scalars c1≈2.15c_1 \approx 2.15c1​≈2.15 and c2≈0.31c_2 \approx 0.31c2​≈0.31 that achieve this. (Finding these scalars involves solving a system of linear equations, a topic covered in Chapter 3).

Understanding linear combinations and span provides a geometric and algebraic framework for thinking about how vectors relate to each other and the spaces they generate. This foundation is important as we move towards analyzing the independence of vectors and the structure of feature spaces.

© 2026 ApX Machine Learning