In many machine learning models you've encountered, like linear regression or standard neural networks, we define a model structure with a fixed set of parameters (weights, biases). Bayesian approaches typically involve placing prior distributions over these parameters. For instance, in Bayesian linear regression, we might assume $y = \mathbf{w}^T \mathbf{x} + \epsilon$ , and place a Gaussian prior on the weight vector $\mathbf{w}$ . Our uncertainty is about the values of these parameters.

Gaussian Processes (GPs) offer a different perspective. Instead of defining priors over parameters of a specific function form, GPs allow us to define a prior distribution directly over functions themselves. Think about it: what if we could model the underlying function $f(x)$ generating our data as a random variable drawn from some distribution? This is precisely what GPs enable.

From Finite to Infinite Dimensions

You're familiar with multivariate Gaussian distributions, which describe the probability distribution of a finite-dimensional vector $\mathbf{f} = [f_1, f_2, ..., f_n]^T$ . A multivariate Gaussian is characterized by a mean vector $\boldsymbol{\mu}$ and a covariance matrix $\Sigma$ : $\mathbf{f} \sim \mathcal{N}(\boldsymbol{\mu}, \Sigma)$ The covariance matrix $\Sigma$ , where $\Sigma_{ij} = \text{Cov}(f_i, f_j)$ , captures the relationships between the different elements of the vector $\mathbf{f}$ .

Now, imagine we want to define a distribution over functions $f(x)$ , where $x$ can be any point in some input domain (potentially continuous and infinite). A function can be thought of as an infinitely long vector, where each element corresponds to the function's value $f(x)$ at a specific input $x$ . How can we extend the idea of a multivariate Gaussian to this infinite-dimensional setting?

This leads us to the definition of a Gaussian Process.

Formal Definition

A Gaussian Process is formally defined as a collection of random variables, any finite subset of which has a joint Gaussian distribution.

Let $f(x)$ represent the value of the random function at input $x$ . A Gaussian Process prior on $f(x)$ is specified by two functions:

Mean Function $m(x)$ : This function defines the expected value of the function at any input $x$ . $m(x) = \mathbb{E}[f(x)]$
Covariance Function (or Kernel) $k(x, x')$ : This function defines the covariance between the function values at two different input points, $x$ and $x'$ . $k(x, x') = \text{Cov}(f(x), f(x')) = \mathbb{E}[(f(x) - m(x))(f(x') - m(x'))]$

We denote a function $f$ drawn from a GP as: $f(x) \sim \mathcal{GP}(m(x), k(x, x'))$

The core implication of the definition is this: if we choose any finite set of input points $\mathbf{x} = [x_1, x_2, ..., x_n]^T$ , the corresponding vector of function values $\mathbf{f} = [f(x_1), f(x_2), ..., f(x_n)]^T$ will be drawn from a multivariate Gaussian distribution: $\mathbf{f} \sim \mathcal{N}(\boldsymbol{\mu}, K)$ where:

The mean vector $\boldsymbol{\mu}$ has elements $\mu_i = m(x_i)$ .
The covariance matrix $K$ (often called the Gram matrix) has elements $K_{ij} = k(x_i, x_j)$ .

The Role of Mean and Covariance Functions

The mean function $m(x)$ represents our prior belief about the average shape of the function before observing any data. In practice, it's common to assume a zero mean function, $m(x) = 0$ , especially after standardizing the data. This simplifies calculations and places the emphasis on the covariance function to capture the function's structure.

The covariance function $k(x, x')$ is where the magic happens. It defines the properties of the functions drawn from the GP prior. It dictates how similar the function values are expected to be at different input points.

If $x$ and $x'$ are close according to the distance measure implicit in the kernel, $k(x, x')$ will typically be large, implying $f(x)$ and $f(x')$ are highly correlated (similar).
If $x$ and $x'$ are far apart, $k(x, x')$ will be small, implying less correlation.

The choice of kernel encodes our assumptions about the function's characteristics, such as smoothness, periodicity, or stationarity. We will examine different kernel functions in detail in the next section.

Visualizing Functions from a GP Prior

To get a feel for what this "distribution over functions" looks like, we can draw sample functions from a GP prior. We do this by:

Choosing a set of input points $x_1, ..., x_n$ .
Defining a mean function $m(x)$ (e.g., $m(x)=0$ ) and a covariance function $k(x, x')$ (e.g., a Squared Exponential kernel).
Computing the mean vector $\boldsymbol{\mu}$ and the covariance matrix $K$ for these points.
Drawing samples from the multivariate Gaussian $\mathcal{N}(\boldsymbol{\mu}, K)$ .
Plotting these samples as functions (connecting the points $(x_i, f(x_i))$ ).

Each line represents a single function drawn from the specified GP prior distribution. Notice how the functions exhibit smoothness, a property encoded by the chosen RBF kernel.

In essence, the GP defines a probability distribution over an infinite-dimensional function space. This prior captures our beliefs about the function before we see any data points. When we combine this prior with observed data (using Bayes' theorem, as we'll see in GP regression), we obtain a posterior distribution, which is also a Gaussian Process, but one that is updated to reflect the information from the data. This posterior GP allows us to make predictions about the function's value at new points, complete with uncertainty estimates.