Understanding Subspaces

Vector spaces are fundamental structures in linear algebra. Within these spaces, important smaller, self-contained structures exist. Imagine the familiar 3D space, $R^3$ . Inside this space, you can visualize planes and lines that pass through the origin. These aren't just arbitrary collections of points; they possess the same fundamental properties as the larger space they inhabit. These special subsets are called subspaces.

Formally, a subset $W$ of a vector space $V$ is called a subspace if $W$ itself forms a vector space under the same addition and scalar multiplication operations defined on $V$ . While checking all vector space axioms might seem tedious, there's a simpler test. A non-empty subset $W$ of $V$ is a subspace if and only if it satisfies three conditions:

Contains the Zero Vector: The zero vector of $V$ must also be in $W$ . ( $0 \in W$ )
Closure under Addition: For any two vectors $u$ and $v$ in $W$ , their sum $u + v$ must also be in $W$ .
Closure under Scalar Multiplication: For any vector $u$ in $W$ and any scalar $c$ , the scalar multiple $c \cdot u$ must also be in $W$ .

Let's unpack these conditions with some geometric intuition.

Geometric Viewpoint: Lines and Planes Through the Origin

Think about $R^2$ , the standard 2D Cartesian plane.

A line passing through the origin: For example, the set $W$ $W$ of all vectors $(x, y)$ $(x, y)$ such that $y = 2x$ $y = 2 x$ .
- Does it contain the zero vector? Yes, $(0, 0)$ satisfies $0 = 2 \times 0$ .
- Is it closed under addition? Let $u = (x_1, 2x_1)$ and $v = (x_2, 2x_2)$ be two vectors on the line. Their sum is $u + v = (x_1 + x_2, 2x_1 + 2x_2) = (x_1 + x_2, 2(x_1 + x_2))$ . This resulting vector also has its y-component equal to twice its x-component, so it lies on the line. $W$ is closed under addition.
- Is it closed under scalar multiplication? Let $u = (x_1, 2x_1)$ be on the line and $c$ be any scalar. Then $c \cdot u = (c x_1, c (2x_1)) = (c x_1, 2(c x_1))$ . This vector also satisfies the condition $y = 2x$ , so it's on the line. $W$ is closed under scalar multiplication. Since all three conditions hold, the line $y=2x$ is a subspace of $R^2$ .

A line passing through the origin in $R^2$ satisfies the subspace conditions. It contains the origin, and adding any two vectors on the line or scaling a vector on the line results in another vector on the same line.

What about a line not passing through the origin? For example, the set $W'$ of vectors $(x, y)$ such that $y = 2x + 1$ . This set fails the first condition: the zero vector $(0, 0)$ is not in $W'$ because $0 \neq 2(0) + 1$ . Therefore, this line is not a subspace of $R^2$ . The requirement that the zero vector must belong to any subspace is fundamental.
The entire space $R^2$ is also a subspace of itself. It trivially satisfies all conditions.
The set containing only the zero vector $\{0\}$ is also a subspace. This is often called the trivial subspace.

Similarly, in $R^3$ , lines and planes passing through the origin $(0, 0, 0)$ are subspaces. A plane defined by $ax + by + cz = 0$ is a subspace, while a plane $ax + by + cz = d$ (where $d \neq 0$ ) is not, because it doesn't contain the origin.

Algebraic Example

For example, take the vector space $V = R^3$ . Let $W$ be the set of all vectors of the form $(x, y, 0)$ , where $x$ and $y$ are any real numbers. Geometrically, this represents the xy-plane within the 3D space. Let's rigorously check the conditions:

Zero Vector: The zero vector in $R^3$ is $(0, 0, 0)$ . This fits the form $(x, y, 0)$ with $x=0$ and $y=0$ . So, the zero vector is in $W$ .
Closure under Addition: Let $u = (x_1, y_1, 0)$ and $v = (x_2, y_2, 0)$ be two arbitrary vectors in $W$ . Their sum is $u + v = (x_1 + x_2, y_1 + y_2, 0 + 0) = (x_1 + x_2, y_1 + y_2, 0)$ . The result is still a vector with a zero in the third component, confirming that $u + v$ is also in $W$ .
Closure under Scalar Multiplication: Let $u = (x_1, y_1, 0)$ be in $W$ and $c$ be any scalar (a real number). Then $c \cdot u = (c x_1, c y_1, c \cdot 0) = (c x_1, c y_1, 0)$ . This resulting vector also has a zero in the third component, so $c \cdot u$ is in $W$ .

Since all three conditions are met, the set $W$ (the xy-plane) is indeed a subspace of $R^3$ .

Why Subspaces Matter in Machine Learning

Understanding subspaces is not just a theoretical exercise; it provides valuable insights into data structure and algorithm behavior:

Feature Subspaces: Imagine your dataset has many features (e.g., hundreds of measurements for each data point), meaning your data vectors live in a high-dimensional space $R^n$ . If you perform feature selection or feature engineering, you might choose to work with a smaller set of features. The data, when represented using only this subset of features, effectively resides within a subspace of the original feature space. The properties of this subspace, such as its dimension (which we'll discuss soon), relate directly to the complexity and redundancy of the selected features. For instance, Principal Component Analysis (PCA) explicitly seeks a lower-dimensional subspace that captures most of the variance in the data.
Solution Spaces: When solving systems of linear equations, particularly homogeneous systems of the form $Ax = 0$ , the set of all possible solution vectors $x$ forms a subspace. This is known as the null space (or kernel) of the matrix $A$ . Understanding this subspace is important for analyzing the uniqueness of solutions to linear systems that arise in model fitting, like in linear regression.
Manifold Hypothesis: While often dealing with non-linear structures, the manifold hypothesis in machine learning suggests that complex, high-dimensional data (like images or natural language embeddings) often lies concentrated near a lower-dimensional manifold embedded within the higher-dimensional ambient space. Although manifolds are generally curved, their local structure can sometimes be approximated by flat subspaces (tangent spaces). Subspaces provide the essential linear algebraic foundation for thinking about data that exhibits lower-dimensional structure, even if that structure isn't perfectly flat globally.

Identifying whether a given set of vectors forms a subspace involves verifying the three closure properties. If any one property fails (e.g., the set doesn't contain the zero vector, or adding two vectors in the set gives a vector outside the set), then it is not a subspace. This systematic check helps us categorize and analyze structured subsets within larger vector spaces, which is fundamental for understanding data transformations, dimensionality reduction techniques, and the behavior of various machine learning models.

Build LLM apps faster with Kerb

Cleaner syntax. Built-in debugging. Production-ready from day one.

Built for the AI systems behind ApX Machine Learning

Was this section helpful?

References

Introduction to Linear Algebra, Gilbert Strang, 2016 (Wellesley-Cambridge Press) - Classic textbook providing a thorough treatment of vector spaces and subspaces, with strong geometric intuition and applications.
Linear Algebra and Its Applications, David C. Lay, Steven R. Lay, Judi J. McDonald, 2021 (Pearson) - Widely-used textbook that clearly explains subspaces with numerous examples and a focus on practical applications.
Mathematics for Machine Learning, Marc Peter Deisenroth, A. Aldo Faisal, Cheng Soon Ong, 2020 (Cambridge University Press) - Provides a focused introduction to linear algebra concepts, including subspaces, as they relate to machine learning algorithms and data analysis.
18.06SC Linear Algebra, Gilbert Strang, 2011 MIT OpenCourseWare (Massachusetts Institute of Technology) - MIT's free online course, featuring lectures and materials that thoroughly cover vector spaces and the properties of subspaces.