Beyond adding vectors or scaling them, another essential operation is the dot product (also known as the scalar product or inner product). Unlike addition or scalar multiplication which result in another vector, the dot product takes two vectors and returns a single scalar number. This operation is fundamental in many areas, including measuring the similarity between vectors and forming the basis of more complex operations like matrix multiplication.

Calculating the Dot Product

The dot product is defined for two vectors of the same dimension. If you have two vectors, say $a$ and $b$ , both with $n$ elements:

a = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix}, \quad b = \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix}

Their dot product, often denoted as $a \cdot b$ or $a^T b$ , is calculated by multiplying corresponding elements and summing the results:

a \cdot b = a_1 b_1 + a_2 b_2 + \dots + a_n b_n = \sum_{i=1}^{n} a_i b_i

Example:

Let's calculate the dot product of two 3-dimensional vectors: $v = [1, 2, 3]$ $w = [4, -5, 6]$

Using the formula: $v \cdot w = (1 \times 4) + (2 \times -5) + (3 \times 6)$ $v \cdot w = 4 - 10 + 18$ $v \cdot w = 12$

The result, 12, is a scalar. Notice that the vectors must have the same number of elements for the dot product to be defined. You cannot compute the dot product of a 3-element vector and a 4-element vector.

Dot Product in Python using NumPy

NumPy provides convenient ways to compute the dot product.

Using the numpy.dot() function:

import numpy as np

v = np.array([1, 2, 3])
w = np.array([4, -5, 6])

# Calculate the dot product using np.dot()
dot_product = np.dot(v, w)

print(f"Vector v: {v}")
print(f"Vector w: {w}")
print(f"Dot product (using np.dot): {dot_product}")

Output:

Vector v: [1 2 3]
Vector w: [ 4 -5  6]
Dot product (using np.dot): 12

Using the @ operator: Python 3.5+ introduced the @ operator for matrix multiplication, which also works for calculating the dot product of 1D arrays (vectors).

import numpy as np

v = np.array([1, 2, 3])
w = np.array([4, -5, 6])

# Calculate the dot product using the @ operator
dot_product_operator = v @ w

print(f"Vector v: {v}")
print(f"Vector w: {w}")
print(f"Dot product (using @): {dot_product_operator}")

Output:

Vector v: [1 2 3]
Vector w: [ 4 -5  6]
Dot product (using @): 12

Both methods yield the same scalar result, 12, matching our manual calculation. The @ operator is often preferred for its conciseness in complex expressions.

Geometric Interpretation: Angles and Projections

The dot product has a powerful geometric interpretation related to the angle between the two vectors. The formula is:

a \cdot b = \|a\| \|b\| \cos(\theta)

Where:

$a \cdot b$ is the dot product of vectors $a$ and $b$ .
$\|a\|$ and $\|b\|$ are the magnitudes (or $L_2$ norms) of vectors $a$ and $b$ .
$\theta$ is the angle between the two vectors.

This formula allows us to find the angle between two vectors if we know their components:

\cos(\theta) = \frac{a \cdot b}{\|a\| \|b\|}

From this, we can infer relationships between vectors based on their dot product:

If $a \cdot b > 0$ : The angle $\theta$ between the vectors is less than 90 degrees ( $\cos(\theta) > 0$ ). The vectors point in a generally similar direction.
If $a \cdot b = 0$ : The angle $\theta$ is exactly 90 degrees ( $\cos(\theta) = 0$ ), meaning the vectors are orthogonal (perpendicular). This is a very important property used throughout linear algebra and machine learning. Note that this assumes $a$ and $b$ are non-zero vectors.
If $a \cdot b < 0$ : The angle $\theta$ is greater than 90 degrees ( $\cos(\theta) < 0$ ). The vectors point in generally opposite directions.

The dot product is also related to the concept of projection. The scalar projection of vector $a$ onto vector $b$ (how much of $a$ points in the direction of $b$ ) can be calculated using the dot product: $\frac{a \cdot b}{\|b\|}$ .

Why is the Dot Product Important in Machine Learning?

The dot product appears frequently in machine learning algorithms:

Similarity Measurement: Cosine similarity, calculated as $\frac{a \cdot b}{\|a\| \|b\|}$ , uses the dot product to measure the similarity in orientation between two vectors, irrespective of their magnitudes. This is widely used in natural language processing (NLP) for comparing document or word vectors and in recommendation systems.
Linear Models: In models like linear regression or logistic regression, the prediction is often calculated as a weighted sum of input features. This weighted sum is precisely a dot product between the feature vector and the model's weight vector.
Neural Networks: The core computation within many neural network layers involves matrix multiplications, which are essentially composed of many dot products.

Understanding the dot product calculation and its geometric meaning provides a foundation for comprehending these more advanced applications. In the next section, we'll practice implementing these vector operations using NumPy.