Expected Value and Variance

Random variables represent numerical values assigned to outcomes from a sample space. To understand and analyze these variables, it is important to summarize their characteristics beyond simply listing possible values. Describing the 'center' and 'spread' of the distribution of these values provides a more complete picture. Expected value and variance are the main measures that provide this summary.

Expected Value: The Center of Mass

The expected value, denoted as $E[X]$ or sometimes $\mu_X$ (or simply $\mu$), represents the weighted average of the possible values a random variable $X$ can take, where the weights are the probabilities of those values. Intuitively, if you were to repeat an experiment involving $X$ many times and calculate the average of the outcomes, that average would converge to the expected value $E[X]$. It's like the "center of mass" of the probability distribution.

Calculating Expected Value

The calculation differs slightly for discrete and continuous random variables:

• For a Discrete Random Variable $X$: If $X$ can take values $x_1, x_2, ..., x_n$ with corresponding probabilities $P(X=x_1), P(X=x_2), ..., P(X=x_n)$, the expected value is: $E[X] = \sum_{i} x_i P(X=x_i)$ The sum is taken over all possible values $x_i$.

• For a Continuous Random Variable $X$: If $X$ has a probability density function (PDF) $f(x)$, the expected value is calculated by integrating the product of $x$ and $f(x)$ over the entire range of $X$: $E[X] = \int_{-\infty}^{\infty} x f(x) dx$

Example: Fair Six-Sided Die

Let $X$ be the random variable representing the outcome of rolling a fair six-sided die. The possible values are $\{1, 2, 3, 4, 5, 6\}$, and each has a probability of $1/6$.

The expected value is: $E[X] = 1 \cdot \frac{1}{6} + 2 \cdot \frac{1}{6} + 3 \cdot \frac{1}{6} + 4 \cdot \frac{1}{6} + 5 \cdot \frac{1}{6} + 6 \cdot \frac{1}{6}$ $E[X] = \frac{1+2+3+4+5+6}{6} = \frac{21}{6} = 3.5$ Notice that the expected value (3.5) is not a value the die can actually land on. It's the long-term average outcome over many rolls.

Properties of Expected Value

Expected value has some useful linear properties:

1. Constants: If $c$ is a constant, then $E[c] = c$.
2. Scaling and Shifting: For constants $a$ and $b$, $E[aX + b] = aE[X] + b$.
3. Sum of Random Variables: For any two random variables $X$ and $Y$, $E[X + Y] = E[X] + E[Y]$. This holds regardless of whether $X$ and $Y$ are independent.

These properties are extremely useful for simplifying calculations involving combinations of random variables.

Variance and Standard Deviation: Measuring Spread

While expected value tells us about the center of a distribution, it doesn't tell us how spread out the values are. Are the values tightly clustered around the mean, or are they widely dispersed? Variance measures this spread.

The variance of a random variable $X$, denoted as $Var(X)$ or $\sigma^2_X$ (or simply $\sigma^2$), is the expected value of the squared difference between the random variable and its expected value $\mu = E[X]$.

$Var(X) = E[(X - \mu)^2]$

A higher variance means the values of $X$ tend to be further away from the mean, on average. A lower variance means they tend to be closer to the mean.

Calculating Variance

Similar to expected value, the calculation depends on whether the variable is discrete or continuous:

• For a Discrete Random Variable $X$: Using the definition $\mu = E[X]$: $Var(X) = \sum_{i} (x_i - \mu)^2 P(X=x_i)$

• For a Continuous Random Variable $X$: Using the definition $\mu = E[X]$ and PDF $f(x)$: $Var(X) = \int_{-\infty}^{\infty} (x - \mu)^2 f(x) dx$

There's often a more convenient computational formula derived from the definition: $Var(X) = E[X^2] - (E[X])^2$ To use this, you first calculate $E[X]$ (the mean) and $E[X^2]$ (the expected value of $X$ squared), then plug them into the formula. Remember, $E[X^2] = \sum_i x_i^2 P(X=x_i)$ for discrete variables and $E[X^2] = \int_{-\infty}^{\infty} x^2 f(x) dx$ for continuous variables.

Standard Deviation

The variance is measured in squared units of the original random variable (e.g., if $X$ is in meters, $Var(X)$ is in meters squared). This can be hard to interpret directly. The standard deviation, denoted as $\sigma_X$ or $SD(X)$ (or simply $\sigma$), is the positive square root of the variance:

$\sigma = \sqrt{Var(X)}$

The standard deviation is measured in the same units as the original random variable $X$, making it more intuitive to understand the typical deviation from the mean.

Example: Fair Six-Sided Die (Continued)

We found $E[X] = 3.5$. Let's calculate the variance using the definition:

$$\begin{aligned} Var(X) = & (1 - 3.5)^2 \frac{1}{6} + (2 - 3.5)^2 \frac{1}{6} + (3 - 3.5)^2 \frac{1}{6} \\ & + (4 - 3.5)^2 \frac{1}{6} + (5 - 3.5)^2 \frac{1}{6} + (6 - 3.5)^2 \frac{1}{6} \\ = & \frac{1}{6} [(-2.5)^2 + (-1.5)^2 + (-0.5)^2 + (0.5)^2 + (1.5)^2 + (2.5)^2] \\ = & \frac{1}{6} [6.25 + 2.25 + 0.25 + 0.25 + 2.25 + 6.25] \\ = & \frac{1}{6} [17.5] \approx 2.917 \end{aligned}$$

Alternatively, using the computational formula $Var(X) = E[X^2] - (E[X])^2$: First, calculate $E[X^2]$:

$$\begin{aligned} E[X^2] &= 1^2 \frac{1}{6} + 2^2 \frac{1}{6} + 3^2 \frac{1}{6} + 4^2 \frac{1}{6} + 5^2 \frac{1}{6} + 6^2 \frac{1}{6} \\ &= \frac{1+4+9+16+25+36}{6} = \frac{91}{6} \approx 15.167 \end{aligned}$$

Now, calculate the variance: $Var(X) = E[X^2] - (E[X])^2 = \frac{91}{6} - (3.5)^2 = \frac{91}{6} - 12.25 = \frac{91}{6} - \frac{73.5}{6} = \frac{17.5}{6} \approx 2.917$ Both methods yield the same result.

The standard deviation is: $\sigma = \sqrt{Var(X)} = \sqrt{\frac{17.5}{6}} \approx \sqrt{2.917} \approx 1.708$ So, for a fair die roll, the expected outcome is 3.5, and the outcomes typically deviate from this mean by about 1.708.

Properties of Variance

Variance also has important properties:

1. Non-negativity: $Var(X) \ge 0$. Variance is zero only if $X$ is a constant.
2. Constants: If $c$ is a constant, $Var(c) = 0$.
3. Scaling and Shifting: For constants $a$ and $b$, $Var(aX + b) = a^2 Var(X)$. Note that adding a constant $b$ shifts the distribution but doesn't change its spread, so $b$ does not affect the variance. Scaling by $a$ scales the variance by $a^2$.
4. Sum of Independent Random Variables: If $X$ and $Y$ are independent random variables, then $Var(X + Y) = Var(X) + Var(Y)$. If they are not independent, the formula involves covariance, a concept we might touch upon later.

Understanding expected value and variance is fundamental. They provide concise summaries of a probability distribution's central tendency and dispersion, forming the basis for many concepts in statistics and machine learning, from evaluating estimators to understanding uncertainty in predictions. In later sections, we'll see how Python libraries like NumPy and SciPy make calculating these values straightforward for various distributions.