In machine learning, we work with data. Lots of data. This data could be anything from customer purchase histories and sensor readings to images and text documents. To apply mathematical algorithms to this diverse data, we first need a consistent way to represent it. This is where vectors come in.

Think of a vector as an ordered list of numbers. Each number in this list represents a specific characteristic or feature of a single data point. By organizing these features into a vector, we create a mathematical object that algorithms can process.

From Data Points to Feature Vectors

Imagine you have data about houses, and you want to predict their prices. For each house, you might record several features:

Size (square feet)
Number of bedrooms
Age (years)
Distance to the nearest school (miles)

A specific house, say House A, might have the following characteristics: 1500 sq ft, 3 bedrooms, 20 years old, and 0.5 miles from a school. We can represent House A as a vector:

$\mathbf{v}_A = \begin{bmatrix} 1500 \\ 3 \\ 20 \\ 0.5 \end{bmatrix}$

This column vector $\mathbf{v}_A$ is often called a feature vector. Each element corresponds to a specific feature in a predefined order. Another house, House B (2100 sq ft, 4 bedrooms, 5 years old, 1.2 miles from school), would be represented by a different vector:

$\mathbf{v}_B = \begin{bmatrix} 2100 \\ 4 \\ 5 \\ 1.2 \end{bmatrix}$

This representation is incredibly versatile:

Images: A grayscale image can be represented as a vector where each element is the intensity value of a pixel. A 100x100 pixel image becomes a vector with 10,000 elements (often by "flattening" the 2D pixel grid into a long 1D list). Color images might have three values (Red, Green, Blue) per pixel, leading to even larger vectors.
Text: Documents can be converted into vectors using techniques like Bag-of-Words (where each element counts the occurrences of a specific word in the vocabulary) or more advanced methods like Word Embeddings (where words or documents are mapped to dense vectors capturing semantic meaning).
Sensor Data: Time-series data from sensors can be represented as vectors where each element is a reading at a specific time point, or a collection of readings from different sensors at one time point.

The Geometric Perspective: Points in Feature Space

Representing data as vectors allows us to think geometrically. If a data point has two features (like height and weight), we can represent it as a 2-dimensional vector. We can visualize this vector as an arrow starting from the origin (0, 0) and ending at the point defined by the feature values (height, weight) in a 2D plane.

Two data points, A=(2, 5) and B=(4.5, 3), represented as vectors originating from the origin in a 2-dimensional feature space.

If we have three features, each data point becomes a vector in 3D space. For data with $n$ features (like our house example with $n=4$ ), each data point corresponds to a vector in an $n$ -dimensional space, often called the feature space. While we can't easily visualize spaces beyond three dimensions, the mathematical principles remain the same.

This vector representation is the foundation upon which most machine learning algorithms are built. It provides a structured format that allows us:

To apply mathematical operations: We can add vectors, scale them, and measure angles and distances between them, which are essential operations in algorithms like k-Nearest Neighbors, Support Vector Machines, and Linear Regression.
To handle high-dimensional data: Linear algebra provides tools to work efficiently even when vectors have thousands or millions of dimensions (features).
To perform data transformations: As we'll see later, matrices can be used to transform these vectors, enabling techniques like dimensionality reduction (PCA) or feature engineering.

In the following sections, we will explore the fundamental operations you can perform on these vectors and how these operations are relevant to machine learning tasks.