Vector Norms: Measuring Length

In machine learning, we often need to quantify the "size" or "magnitude" of a vector. While we intuitively understand length in 2D or 3D space, vectors representing data features can exist in much higher dimensions. We need a consistent way to measure this length. This is where the concept of a norm comes in. A norm is a function that takes a vector as input and returns a non-negative scalar value representing its magnitude. Think of it as a formal generalization of the concept of length.

Different norms measure length in different ways, and the choice of norm can have significant implications in machine learning algorithms, particularly in areas like regularization and error calculation. Let's explore the most common norms you'll encounter.

The $L_2$ Norm (Euclidean Norm)

The most familiar norm is the $L_2$ norm, also known as the Euclidean norm. It corresponds to the standard geometric length of a vector, calculated as the square root of the sum of the squares of its components. For an n-dimensional vector $v = [v_1, v_2, ..., v_n]$, the $L_2$ norm is defined as:

$||v||_2 = \sqrt{v_1^2 + v_2^2 + ... + v_n^2} = \sqrt{\sum_{i=1}^{n} v_i^2}$

This is essentially the Pythagorean theorem generalized to higher dimensions. It represents the shortest straight-line distance from the origin to the point represented by the vector.

Example: Consider the vector $v = [3, 4]$. Its $L_2$ norm is: $||v||_2 = \sqrt{3^2 + 4^2} = \sqrt{9 + 16} = \sqrt{25} = 5$

import numpy as np

v = np.array([3, 4])
l2_norm = np.linalg.norm(v) # Default norm is L2

print(f"Vector: {v}")
print(f"L2 Norm: {l2_norm}")
# Output:
# Vector: [3 4]
# L2 Norm: 5.0

The $L_2$ norm measures the direct Euclidean distance from the origin (0,0) to the point (3,4).

In machine learning, the $L_2$ norm is frequently used:

• Calculating Euclidean distances between data points (covered in the next section).
• Measuring the magnitude of error vectors.
• In regularization techniques like Ridge Regression, where it penalizes large coefficient values to prevent overfitting.

The $L_1$ Norm (Manhattan Norm)

Another important norm is the $L_1$ norm, often called the Manhattan norm or Taxicab norm. Instead of squaring components, it sums their absolute values:

$||v||_1 = |v_1| + |v_2| + ... + |v_n| = \sum_{i=1}^{n} |v_i|$

The name "Manhattan norm" comes from the idea of navigating a grid-like city plan. You can only travel along the grid lines (like city blocks), not diagonally. The $L_1$ norm represents the total distance traveled along these axes to get from the origin to the vector's endpoint.

Example: For the same vector $v = [3, 4]$. Its $L_1$ norm is: $||v||_1 = |3| + |4| = 3 + 4 = 7$

import numpy as np

v = np.array([3, 4])
l1_norm = np.linalg.norm(v, ord=1)

print(f"Vector: {v}")
print(f"L1 Norm: {l1_norm}")
# Output:
# Vector: [3 4]
# L1 Norm: 7.0

The $L_1$ norm measures the sum of the absolute values of the components, like moving along grid lines from (0,0) to (3,0) and then to (3,4).

The $L_1$ norm has distinct properties useful in machine learning:

• It's less sensitive to outliers compared to the $L_2$ norm because it doesn't square the component values.
• It often promotes sparsity when used in regularization (like LASSO regression). This means it tends to push less important feature coefficients towards exactly zero, effectively performing feature selection.

Comparing $L_1$ and $L_2$ Norms

The key difference lies in how they treat component magnitudes. The $L_2$ norm squares values, heavily penalizing large components. The $L_1$ norm takes absolute values, treating deviations linearly.

This difference is visually apparent when considering all vectors with a norm equal to 1. For the $L_2$ norm, this forms a circle (in 2D) or a hypersphere (in higher dimensions). For the $L_1$ norm, it forms a diamond (in 2D) or a cross-polytope (in higher dimensions).

Geometric shapes representing vectors where $||v||_2 = 1$ (blue circle) and $||v||_1 = 1$ (red diamond) in two dimensions. The "corners" of the $L_1$ diamond correspond to the axes, contributing to its tendency to favor sparse solutions (where some components are exactly zero) in optimization problems.

Other Norms ($L_p$ and $L_\infty$)

The $L_1$ and $L_2$ norms are specific cases of a broader family called $L_p$ norms, defined as:

$||v||_p = \left( \sum_{i=1}^{n} |v_i|^p \right)^{1/p}$

where $p \ge 1$.

Another norm sometimes encountered is the $L_\infty$ norm (or max norm), which is the limit of the $L_p$ norm as $p$ approaches infinity. It simply returns the maximum absolute value among the vector's components:

$||v||_\infty = \max_i |v_i|$

While $L_1$ and $L_2$ are the workhorses in many machine learning applications, being aware of the general $L_p$ family provides broader context.

Why Norms Matter in Practice

Understanding vector norms is fundamental because they provide ways to measure:

• Magnitude of Features: How "strong" is a particular feature vector?
• Error Measurement: How large is the difference between predicted and actual values (often represented as an error vector)? Different norms for error calculation lead to different model behaviors (e.g., minimizing mean squared error ($L_2$) vs. mean absolute error ($L_1$)).
• Regularization: Norms are used to penalize complexity in models. $L_2$ regularization (Ridge) shrinks coefficients, while $L_1$ regularization (LASSO) can shrink coefficients to zero, performing feature selection.
• Distance Metrics: As we'll see next, norms are the foundation for calculating distances between vectors, which is essential for algorithms like k-Nearest Neighbors (k-NN).

Being able to calculate these norms efficiently, often using libraries like NumPy, is a core skill for implementing and understanding many machine learning algorithms.