Representing Data with Vectors and Matrices

Working with data is fundamental in machine learning. Consider a simple spreadsheet or a database table. Each row might represent a specific item, like a customer, a house, or an image, while each column represents a characteristic or 'feature' of that item, such as age, size, price, or pixel intensity. Linear algebra provides a powerful and efficient way to represent and manipulate this kind of structured data using vectors and matrices.

Representing Individual Data Points as Vectors

Let's start with a single data point. Imagine you're collecting data about houses to predict their prices. For one specific house, you might record:

• Size (square feet)
• Number of bedrooms
• Age (years)
• Distance to the nearest school (miles)

We can represent this single house as an ordered list of numbers, which is precisely what a vector is in linear algebra. If the house has 1500 sq ft, 3 bedrooms, is 10 years old, and is 0.5 miles from a school, we can represent it as the vector $\mathbf{x}$:

$$\mathbf{x} = \begin{bmatrix} 1500 \\ 3 \\ 10 \\ 0.5 \end{bmatrix}$$

Here, $\mathbf{x}$ is a column vector. Each number in the vector corresponds to one of the features we measured. The order matters; the first element always represents size, the second always represents the number of bedrooms, and so on. This vector $\mathbf{x}$ now encapsulates all the measured information about that specific house in a single mathematical object.

In machine learning terminology:

• The entire vector $\mathbf{x}$ often represents a single sample or data instance.
• Each number within the vector (1500, 3, 10, 0.5) is a feature value for that sample.
• The total number of features (4 in this case) determines the dimension of the vector. This is a 4-dimensional vector.

We could also think of a single feature across all houses as a vector. For example, if we had 5 houses in our dataset, the sizes of all 5 houses could form a vector:

$$\text{Sizes} = \begin{bmatrix} 1500 \\ 2100 \\ 1200 \\ 1800 \\ 2400 \end{bmatrix}$$

This vector represents the 'size' feature column from our data table.

Representing Datasets as Matrices

Now, what if we have data for multiple houses? Let's say we have data for three houses:

• House 1: 1500 sq ft, 3 bedrooms, 10 years old, 0.5 miles to school
• House 2: 2100 sq ft, 4 bedrooms, 5 years old, 1.2 miles to school
• House 3: 1200 sq ft, 2 bedrooms, 20 years old, 0.8 miles to school

We can represent each house as a vector, as shown before:

$$\mathbf{x}^{(1)} = \begin{bmatrix} 1500 \\ 3 \\ 10 \\ 0.5 \end{bmatrix}, \quad \mathbf{x}^{(2)} = \begin{bmatrix} 2100 \\ 4 \\ 5 \\ 1.2 \end{bmatrix}, \quad \mathbf{x}^{(3)} = \begin{bmatrix} 1200 \\ 2 \\ 20 \\ 0.8 \end{bmatrix}$$

(Note: We use $\mathbf{x}^{(i)}$ to denote the vector for the $i$-th data sample to avoid confusion with $x_j$ which often denotes the $j$-th feature within a vector).

To represent the entire dataset, we can stack these individual vectors together as columns (or more commonly, as rows) in a grid. This rectangular grid of numbers is called a matrix. If we arrange each house vector as a row in our matrix (a common convention in machine learning), our dataset becomes:

$$X = \begin{bmatrix} 1500 & 3 & 10 & 0.5 \\ 2100 & 4 & 5 & 1.2 \\ 1200 & 2 & 20 & 0.8 \end{bmatrix}$$

This matrix $X$ neatly organizes our entire dataset.

In this matrix:

• Each row corresponds to a single data sample (a house).
• Each column corresponds to a single feature (size, bedrooms, age, distance).
• The dimensions of the matrix tell us the size of the dataset. This matrix has 3 rows and 4 columns, so its dimensions are $3 \times 4$. We say it's a "3 by 4" matrix. In general, an $m \times n$ matrix represents $m$ data samples, each having $n$ features.

This matrix representation is fundamental. It allows us to apply mathematical operations (which we'll cover in later chapters) to the entire dataset simultaneously. Operations like calculating averages for each feature, transforming features, or feeding the data into a machine learning model often rely on this matrix structure. Libraries like NumPy in Python are specifically designed to work efficiently with these vector and matrix representations of data.