Point Estimation

Statistical inference allows making educated guesses about a larger population using only data from a smaller sample. One of the most direct ways to do this is through point estimation.

The core idea is simple: We want to estimate an unknown characteristic of the population, called a parameter. Since we usually can't measure the entire population, we calculate a similar characteristic from our sample, called a statistic. This sample statistic serves as our single "best guess," or point estimate, for the unknown population parameter.

Population Parameters vs. Sample Statistics

It's important to distinguish between these two concepts:

• Population Parameter: A numerical value summarizing some characteristic of the entire population. These are typically unknown and what we want to estimate. Examples include:
  • The true average height of all adult women in Canada (population mean, often denoted by $\mu$).
  • The true proportion of all website visitors who click on a specific ad (population proportion, often denoted by $p$).
  • The true variability in the diameter of parts produced by a machine (population variance, $\sigma^2$, or standard deviation, $\sigma$).
• Sample Statistic: A numerical value calculated from sample data. We use this value to estimate the corresponding population parameter. Examples include:
  • The average height calculated from a sample of 500 Canadian women (sample mean, denoted by $\bar{x}$).
  • The proportion of clicks observed in a sample of 10,000 website visitors (sample proportion, denoted by $\hat{p}$).
  • The variability calculated from a sample of 100 parts (sample variance, $s^2$, or standard deviation, $s$).

Think of the population parameter as the fixed, true value we're aiming for, and the sample statistic as our calculated approximation based on the data we could collect.

Common Point Estimators

The formula or rule we use to calculate the sample statistic is called an estimator. The specific value we get after applying the estimator to our sample data is the estimate.

Here are some common examples:

1. Estimating Population Mean ($\mu$): The most common estimator for the population mean $\mu$ is the sample mean, $\bar{X}$. The formula (the estimator) is:

   $$\bar{X} = \frac{\sum_{i=1}^{n} X_i}{n} = \frac{X_1 + X_2 + ... + X_n}{n}$$

   If we collect a sample and calculate the sample mean, say $\bar{x} = 165.2$ cm, then $165.2$ cm is our point estimate of the population mean height $\mu$.

2. Estimating Population Proportion ($p$): To estimate the population proportion $p$ (e.g., proportion of defective items), we use the sample proportion, $\hat{P}$. The estimator is:

   $$\hat{P} = \frac{\text{Number of successes in sample}}{\text{Sample size}} = \frac{x}{n}$$

   If we test 200 items (n=200) and find 10 defectives (x=10), our point estimate for the true proportion of defectives is $\hat{p} = 10 / 200 = 0.05$ or 5%.

3. Estimating Population Variance ($\sigma^2$): A common estimator for the population variance $\sigma^2$ is the sample variance, $S^2$. The formula is:

   $$S^2 = \frac{\sum_{i=1}^{n} (X_i - \bar{X})^2}{n-1}$$

   Notice the $n-1$ in the denominator. Using $n-1$ instead of $n$ makes $S^2$ an "unbiased" estimator of $\sigma^2$, meaning it doesn't systematically underestimate or overestimate the true population variance on average. Calculating this value for our sample gives us the point estimate, $s^2$.

Why is it called a "Point" Estimate?

It's called a point estimate because it provides a single numerical value as the estimate for the population parameter. Imagine a number line representing all possible values for the parameter (like average height); our point estimate is a single point on that line representing our best guess based on the sample.

For example, if we estimate the average customer spending to be $45.50, that single value is our point estimate. While it's our best guess, we know it's unlikely to be exactly equal to the true population average spending. Samples vary, and a different sample would likely produce a slightly different estimate. This inherent uncertainty leads us to the concept of interval estimation (confidence intervals), which we'll discuss next. Interval estimates provide a range of plausible values for the parameter, acknowledging the uncertainty involved.

Example: Estimating Average Order Value

Let's say we run an e-commerce site and want to estimate the average order value (AOV) for all customers (the population parameter $\mu$). We can't calculate it from all orders, so we take a sample of 50 recent orders:

import numpy as np

# Sample of 50 recent order values (in dollars)
order_values = np.array([
    55.2, 34.1, 89.0, 120.5, 42.0, 65.8, 22.1, 98.7, 77.3, 105.6,
    30.9, 48.2, 71.5, 60.0, 112.8, 58.4, 88.2, 45.1, 99.9, 75.0,
    135.2, 28.6, 50.0, 68.9, 91.3, 102.1, 40.5, 79.8, 66.2, 84.7,
    115.0, 33.5, 52.8, 70.1, 95.4, 108.3, 47.9, 62.7, 81.6, 100.2,
    36.8, 56.7, 73.9, 64.3, 89.1, 110.5, 49.6, 67.0, 83.3, 97.4
])

# Calculate the sample mean (our point estimate for mu)
sample_mean_aov = np.mean(order_values)

# Calculate the sample standard deviation (point estimate for sigma)
# Use ddof=1 for the unbiased sample standard deviation (estimates population std dev)
sample_std_dev = np.std(order_values, ddof=1)

print(f"Sample Size (n): {len(order_values)}")
print(f"Point Estimate for Average Order Value (μ): ${sample_mean_aov:.2f}")
print(f"Point Estimate for Standard Deviation (σ): ${sample_std_dev:.2f}")

Sample Size (n): 50
Point Estimate for Average Order Value (μ): $74.29
Point Estimate for Standard Deviation (σ): $27.68

Based on this sample, our point estimate for the true average order value ($\mu$) across all customers is $74.29. Our point estimate for the population standard deviation ($\sigma$) is$27.68.

Histogram of the 50 sample order values. The dashed red line indicates the calculated sample mean ($74.29), which serves as the point estimate for the unknown average order value of all customers.

Point estimates form the foundation of many statistical inference procedures and are frequently used in machine learning, for example, when estimating the error rate of a model based on a test set or determining the parameters of a probability distribution assumed to model the data. However, it's important to remember they represent a single guess based on limited data. Next, we'll explore how to quantify the uncertainty surrounding these estimates.