Probability Density Function (PDF) for Continuous Distributions

Probability Mass Functions (PMFs) describe discrete random variables, for example, the number of heads in three coin flips. For discrete variables, all possible outcomes can be listed, and a specific probability is assigned to each one. The PMF provides $P(X=x)$, the probability that the random variable $X$ takes on the exact value $x$.

But what happens when the variable can take on any value within a continuous range? Think about measuring someone's exact height, the precise temperature, or the time it takes for a process to complete. These are continuous random variables. If we tried to assign a probability to a single, infinitely precise value (like a height of exactly 175.0000... cm), the probability would effectively be zero. There are simply too many possibilities!

Instead of focusing on the probability of a single point, for continuous variables, we talk about the probability of the variable falling within a specific interval. This is where the Probability Density Function (PDF) comes in.

Understanding the Probability Density Function (PDF)

The PDF, often denoted as $f(x)$, is a function that describes the relative likelihood for a continuous random variable to take on a given value. Unlike the PMF, the value of the PDF at a specific point $x$, $f(x)$, is not a probability itself. Instead, the area under the curve of the PDF between two points, say $a$ and $b$, represents the probability that the random variable $X$ falls within that interval $[a, b]$.

Mathematically, this is expressed using an integral:

$P(a \le X \le b) = \int_{a}^{b} f(x) dx$

Don't worry if you haven't seen the $\int$ symbol or calculus before. The important concept is that Area = Probability. For many standard distributions we'll encounter, we won't need to perform the integration manually; we can use tables or software functions.

Think of it like this: imagine a histogram for a very large dataset with continuous values. As you make the bins smaller and smaller, the tops of the bars start to form a smooth curve. This smooth curve represents the PDF. The height of the curve at any point indicates where values are more densely clustered.

Properties of a PDF

A valid PDF must satisfy two main properties:

1. Non-negativity: The density function must always be greater than or equal to zero for all possible values of $x$. You can't have negative likelihood. $f(x) \ge 0 \text{ for all } x$
2. Total Area is 1: Since the random variable must take on some value, the total area under the entire curve of the PDF must equal 1. This represents 100% probability. $\int_{-\infty}^{\infty} f(x) dx = 1$

Interpreting the PDF

It's important to remember:

• The value $f(x)$ represents density, not probability. A higher value of $f(x)$ compared to $f(y)$ means the variable is more likely to be found in a small interval around $x$ than in an equally small interval around $y$. It's possible for $f(x)$ to be greater than 1 for some distributions, as long as the total area under the curve remains 1.
• For a continuous variable $X$, the probability of it taking any single specific value is zero: $P(X = c) = 0$. This is because there's no "area" under the curve at a single point (an interval from $c$ to $c$ has zero width). This also means $P(a \le X \le b)$ is the same as $P(a < X < b)$.

Let's visualize this. Consider a generic PDF curve:

The shaded region represents the probability $P(2 \le X \le 4)$. This probability is equal to the area under the blue curve between $x=2$ and $x=4$. The height of the curve $f(x)$ indicates the density of probability around the value $x$.

PDF vs. PMF: A Quick Comparison

Feature	PMF (Discrete)	PDF (Continuous)
Applies to	Discrete Random Variables	Continuous Random Variables
Function Value	$P(X=x)$ (Probability at a point)	$f(x)$ (Density at a point)
Probability	Sum of PMF values over a set	Area (Integral) under PDF over interval
Value Range	$0 \le P(X=x) \le 1$	$f(x) \ge 0$ (can be > 1)
Sum/Integral	$\sum_{\text{all } x} P(X=x) = 1$	$\int_{-\infty}^{\infty} f(x) dx = 1$

Understanding PDFs is significant when working with continuous measurements in machine learning. Many algorithms assume that features follow certain distributions (like the Normal distribution, which we'll see next), and PDFs allow us to model and work with these continuous quantities effectively. They help us quantify uncertainty and make probabilistic statements about continuous outcomes.