Defining Vector Spaces

Vectors represent individual data points or features (like pixel intensities, user ratings, or sensor measurements), and matrices represent entire datasets or transformations. What are the fundamental rules governing how these vectors behave together? Where do these vectors "live" mathematically? These questions introduce the concept of a Vector Space.

Think of a vector space as a collection of objects, which we call vectors, along with two fundamental operations: vector addition and scalar multiplication. For a collection to qualify as a vector space, these operations must satisfy a specific set of rules, often called axioms. These rules ensure that the operations behave in a consistent and predictable way, much like the rules of arithmetic govern numbers.

Why is this formal definition important? Because it provides a solid mathematical foundation. When we know our data vectors live in a vector space, we automatically know they obey these reliable rules. This allows us to build powerful and general techniques for data analysis and machine learning that work consistently across different types of vector data.

The Vector Space Axioms

Let $V$ be a set of vectors, and let $u, v, w$ be any vectors in $V$. Let $c, d$ be any scalars (usually real numbers in machine learning contexts). For $V$ to be a vector space, the following ten axioms must hold true:

Properties of Vector Addition:

1. Closure under Addition: If $u$ and $v$ are in $V$, then their sum $u + v$ must also be in $V$.

   • Intuition: Adding two vectors from the space keeps you within that space. If you add two feature vectors representing images, the result should still be representable in the same kind of feature vector format.

2. Commutativity of Addition: $u + v = v + u$.

   • Intuition: The order in which you add vectors doesn't matter.

3. Associativity of Addition: $(u + v) + w = u + (v + w)$.

   • Intuition: When adding multiple vectors, the grouping doesn't affect the result.

4. Existence of a Zero Vector: There exists a unique vector $0$ in $V$, called the zero vector, such that $u + 0 = u$ for all $u$ in $V$.

   • Intuition: There's a neutral element for addition, like the number 0 in regular arithmetic. In $R^n$, this is the vector of all zeros.

5. Existence of Additive Inverses: For every vector $u$ in $V$, there exists a unique vector $-u$ in $V$, called the additive inverse, such that $u + (-u) = 0$.

   • Intuition: Every vector has an "opposite" vector that cancels it out when added.

Properties of Scalar Multiplication:

6. Closure under Scalar Multiplication: If $u$ is in $V$ and $c$ is a scalar, then the product $c u$ must also be in $V$.

   • Intuition: Scaling a vector keeps you within the same space. Stretching or shrinking a feature vector results in a vector of the same type.

7. Associativity of Scalar Multiplication: $(cd)u = c(du)$.

   • Intuition: The order of applying multiple scalar multiplications doesn't matter.

Properties Connecting Addition and Scalar Multiplication (Distributive Laws):

8. Distributivity over Vector Sum: $c(u + v) = cu + cv$.

   • Intuition: Scaling a sum of vectors is the same as summing the scaled vectors.

9. Distributivity over Scalar Sum: $(c + d)u = cu + du$.

   • Intuition: Scaling a vector by a sum of scalars is the same as summing the vector scaled by each scalar individually.

10. Scalar Multiplication Identity: $1u = u$, where $1$ is the multiplicative identity scalar.

    • Intuition: Multiplying a vector by the scalar 1 leaves the vector unchanged.

The Canonical Example: $R^n$

The most common vector space you'll encounter in machine learning is $R^n$, the space of all $n$-dimensional vectors with real number components.

For example, consider $R^2$, the space of all vectors of the form $[x, y]$, where $x$ and $y$ are real numbers. This corresponds to the familiar 2D Cartesian plane. Let's quickly check a couple of axioms:

• Closure under Addition: If $u = [u_1, u_2]$ and $v = [v_1, v_2]$ are in $R^2$, then $u + v = [u_1 + v_1, u_2 + v_2]$. Since $u_1+v_1$ and $u_2+v_2$ are real numbers, $u+v$ is also in $R^2$.
• Closure under Scalar Multiplication: If $u = [u_1, u_2]$ is in $R^2$ and $c$ is a real scalar, then $cu = [cu_1, cu_2]$. Since $cu_1$ and $cu_2$ are real numbers, $cu$ is also in $R^2$.
• Zero Vector: The zero vector is $[0, 0]$, which is clearly in $R^2$.
• Additive Inverse: For $u = [u_1, u_2]$, the inverse is $-u = [-u_1, -u_2]$, which is also in $R^2$.

You can verify that $R^2$, and indeed $R^n$ for any positive integer $n$, satisfies all ten axioms using the standard definitions of vector addition and scalar multiplication you learned in Chapter 1.

When we represent data points as feature vectors in $R^n$, we are implicitly working within a vector space. This means we can rely on these properties when manipulating data, building models (like linear regression, which heavily relies on vector addition and scalar multiplication), or analyzing relationships between data points.

Understanding these formal properties is the first step towards exploring more complex concepts like subspaces, linear independence, basis, and dimension, which are essential for analyzing the structure within datasets and the behavior of machine learning algorithms.