Calculating Confidence Intervals for Means

A confidence interval is a range computed from sample data that is likely to contain the true value of an unknown population parameter, such as the population mean ($μ$). Here is how to compute these intervals for the mean.

The general structure for a confidence interval is:

$\text{Point Estimate} \pm \text{Margin of Error}$

For the population mean ($\mu$), our best point estimate is usually the sample mean ($\bar{x}$). The margin of error quantifies the uncertainty around this point estimate and depends on two main factors: the variability of the data and how confident we want to be. Specifically:

$\text{Margin of Error} = \text{Critical Value} \times \text{Standard Error of the Mean}$

Standard Error of the Mean (SEM)

Before we build the full interval, we need to understand the standard error of the mean (SEM). Remember the Central Limit Theorem? It tells us that the distribution of sample means (the sampling distribution) tends to be normal, centered around the true population mean $\mu$. The standard deviation of this sampling distribution is the SEM. It measures how much we expect sample means to vary from sample to sample.

If we know the true population standard deviation $\sigma$, the SEM is: $SE = \frac{\sigma}{\sqrt{n}}$ where $n$ is the sample size.

However, we usually don't know the population standard deviation $\sigma$. So, we estimate it using the sample standard deviation $s$. In this case, the estimated SEM is: $SE = \frac{s}{\sqrt{n}}$ This estimated SEM is what we'll typically use in practice. Notice that as the sample size $n$ increases, the SEM decreases. Larger samples lead to more precise estimates of the population mean.

Calculating the Confidence Interval

The exact calculation depends on whether the population standard deviation ($\sigma$) is known or unknown.

Case 1: Population Standard Deviation ($\sigma$) is Known

"This scenario is less common in machine learning problems because if you knew the population standard deviation, you might already know the population mean, negating the need for an interval. However, it's a simpler starting point."

When $\sigma$ is known, the sampling distribution of the mean follows a normal distribution (thanks to the CLT, assuming $n$ is sufficiently large or the population is normally distributed). We use the standard normal (Z) distribution to find the critical value.

The confidence interval formula is: $\bar{x} \pm Z_{\alpha/2} \frac{\sigma}{\sqrt{n}}$

Here:

• $\bar{x}$ is the sample mean.
• $\sigma$ is the known population standard deviation.
• $n$ is the sample size.
• $Z_{\alpha/2}$ is the critical value from the standard normal distribution. It corresponds to the Z-score that cuts off the upper $\alpha/2$ area in the tail. For a chosen confidence level of $(1-\alpha)$, $\alpha$ is the significance level (e.g., for a 95% confidence interval, $\alpha = 0.05$, and $\alpha/2 = 0.025$).

You can find $Z_{\alpha/2}$ using standard normal tables or Python's SciPy library:

import scipy.stats as stats
import numpy as np

# Example: Find Z critical value for a 95% confidence interval
confidence_level = 0.95
alpha = 1 - confidence_level
# Use ppf (percent point function, inverse of CDF)
# We need the point where 1 - alpha/2 of the distribution is to the left
z_critical = stats.norm.ppf(1 - alpha / 2)

print(f"Z critical value for {confidence_level*100}% confidence: {z_critical:.3f}")
# Output: Z critical value for 95.0% confidence: 1.960

So, for a 95% confidence interval with known $\sigma$, the margin of error is $1.960 \times \frac{\sigma}{\sqrt{n}}$.

Case 2: Population Standard Deviation ($\sigma$) is Unknown

This is the much more common situation. We don't know $\sigma$, so we estimate it using the sample standard deviation $s$. When we substitute $s$ for $\sigma$ in the standard error calculation ($SE = s/\sqrt{n}$), the sampling distribution of the statistic $\frac{\bar{x} - \mu}{s/\sqrt{n}}$ no longer follows a standard normal distribution perfectly, especially for smaller sample sizes.

Instead, it follows a t-distribution (also known as Student's t-distribution). The t-distribution is similar in shape to the normal distribution (bell-shaped, symmetric around zero) but has heavier tails. This means it accounts for the additional uncertainty introduced by estimating $\sigma$ with $s$.

The t-distribution is characterized by its degrees of freedom (df), which depend on the sample size. For a confidence interval for a single mean, the degrees of freedom are: $df = n - 1$

The confidence interval formula when $\sigma$ is unknown is: $\bar{x} \pm t_{\alpha/2, n-1} \frac{s}{\sqrt{n}}$

Here:

• $\bar{x}$ is the sample mean.
• $s$ is the sample standard deviation.
• $n$ is the sample size.
• $t_{\alpha/2, n-1}$ is the critical value from the t-distribution with $n-1$ degrees of freedom. It cuts off the upper $\alpha/2$ area in the tail for a confidence level of $(1-\alpha)$.

As the sample size $n$ (and thus the degrees of freedom) increases, the t-distribution approaches the standard normal distribution.

You can find the t-critical value using SciPy:

import scipy.stats as stats
import numpy as np

# Example: Find t critical value for a 95% CI with n=25
sample_size = 25
confidence_level = 0.95
alpha = 1 - confidence_level
degrees_freedom = sample_size - 1

# Use ppf for the t-distribution
t_critical = stats.t.ppf(1 - alpha / 2, df=degrees_freedom)

print(f"t critical value for {confidence_level*100}% confidence (df={degrees_freedom}): {t_critical:.3f}")
# Output: t critical value for 95.0% confidence (df=24): 2.064

Notice this t-critical value (2.064) is slightly larger than the Z-critical value (1.960) for the same confidence level, resulting in a wider interval, reflecting the added uncertainty.

Example Calculation (σ Unknown)

Let's say we're evaluating the inference time (in milliseconds) of a machine learning model. We run the model on a sample of $n=30$ different inputs and record the times. We find:

• Sample mean ($\bar{x}$) = 85 ms
• Sample standard deviation ($s$) = 12 ms

We want to calculate a 99% confidence interval for the true average inference time ($\mu$).

1. Confidence Level and Alpha: Confidence level = 0.99, so $\alpha = 1 - 0.99 = 0.01$. $\alpha/2 = 0.005$.
2. Degrees of Freedom: $df = n - 1 = 30 - 1 = 29$.
3. Find Critical t-value: We need $t_{0.005, 29}$.

   import scipy.stats as stats
   t_critical = stats.t.ppf(1 - 0.01 / 2, df=29)
   print(f"t_critical: {t_critical:.3f}")
   # Output: t_critical: 2.756
   

4. Calculate Standard Error: $SE = \frac{s}{\sqrt{n}} = \frac{12}{\sqrt{30}} \approx \frac{12}{5.477} \approx 2.191$ ms.
5. Calculate Margin of Error: $ME = t_{\alpha/2, n-1} \times SE = 2.756 \times 2.191 \approx 6.039$ ms.
6. Construct the Interval: $\bar{x} \pm ME = 85 \pm 6.039$ The interval is $(85 - 6.039, 85 + 6.039) = (78.961, 91.039)$ ms.

Interpretation: We are 99% confident that the true average inference time for this model across all possible inputs lies between 78.96 ms and 91.04 ms, based on our sample data.

Using SciPy for Direct Calculation

SciPy's stats.t.interval function can compute the interval directly:

import scipy.stats as stats
import numpy as np

# Sample data parameters
sample_mean = 85
sample_std_dev = 12
sample_size = 30
confidence = 0.99

# Calculate standard error
sem = sample_std_dev / np.sqrt(sample_size)
degrees_freedom = sample_size - 1

# Calculate the confidence interval
ci = stats.t.interval(confidence, degrees_freedom, loc=sample_mean, scale=sem)

print(f"Sample Mean: {sample_mean}")
print(f"Standard Error: {sem:.3f}")
print(f"{confidence*100}% Confidence Interval: ({ci[0]:.3f}, {ci[1]:.3f})")
# Output:
# Sample Mean: 85
# Standard Error: 2.191
# 99.0% Confidence Interval: (78.961, 91.039)

This confirms our manual calculation.

Factors Influencing Confidence Interval Width

The width of the confidence interval (Upper Limit - Lower Limit) reflects the precision of our estimate. Two primary factors influence this width:

1. Confidence Level: A higher confidence level (e.g., 99% vs. 95%) requires a larger critical value ($Z_{\alpha/2}$ or $t_{\alpha/2, n-1}$), leading to a wider interval. We need a wider range to be more confident it captures the true mean.
2. Sample Size: A larger sample size ($n$) decreases the standard error ($s/\sqrt{n}$). This results in a smaller margin of error and a narrower, more precise interval.

The plot illustrates how the relative width of a confidence interval (proportional to Critical Value / sqrt(n)) decreases rapidly as sample size increases. Higher confidence levels consistently produce wider intervals for any given sample size. (Note: Uses Z-values for simplicity, t-values show similar behavior).

Calculating confidence intervals provides a practical way to quantify the uncertainty in our estimates based on sample data, moving to simple point estimates to provide a range of plausible values for population parameters like the mean. This is a foundational technique for drawing reliable conclusions from data.