Basics of Probability

Probability is a mathematical field that allows us to quantify and manage uncertainty. In machine learning, where data is inherently noisy and unpredictable, understanding probability becomes crucial. This section will introduce you to the fundamentals of probability, equipping you with the foundational tools necessary for more advanced statistical models and algorithms.

Defining Probability

At its core, probability is a measure of the likelihood of an event occurring. It is quantified as a number between 0 and 1, where 0 indicates impossibility and 1 indicates certainty. Formally, the probability of an event $A$ is denoted by $P(A)$.

The Axioms of Probability

Probability theory is built upon three fundamental axioms introduced by the Russian mathematician Andrey Kolmogorov:

1. Non-negativity: For any event $A$, $P(A) \geq 0$.
2. Normalization: The probability of a certain event, the sample space $S$, is 1, i.e., $P(S) = 1$.
3. Additivity: For any two mutually exclusive events $A$ and $B$, the probability of either event occurring is the sum of their individual probabilities, i.e., $P(A \cup B) = P(A) + P(B)$.

These axioms provide a rigorous framework for reasoning about uncertainty and form the basis for all probability calculations.

Calculating Probabilities

Calculating probabilities often involves understanding the context and characteristics of the events. For instance, in a discrete setting, where outcomes can be counted, the probability of an event is the ratio of the number of favorable outcomes to the total number of possible outcomes. In contrast, continuous settings require integrating probability density functions over a range of values.

Discrete probability calculation as a ratio of favorable to total outcomes.

Random Variables and Distributions

A random variable is a variable that can take on different values, each with an associated probability. Random variables can be discrete, like the roll of a dice, or continuous, like measuring temperature. The probability distribution of a random variable describes how probabilities are distributed over the values of the random variable. Common probability distributions you will encounter include:

• Binomial Distribution: Models the number of successes in a fixed number of independent Bernoulli trials. It is characterized by two parameters: the number of trials $n$ and the probability of success $p$.

Binomial distribution for n=10 trials and p=0.3 probability of success.

• Poisson Distribution: Models the number of events occurring in a fixed interval of time or space, where events occur with a constant mean rate and independently of the time since the last event.

Poisson distribution with mean rate λ=3.

• Normal Distribution: Also known as the Gaussian distribution, it is a continuous distribution characterized by its symmetric bell shape. It is defined by two parameters: the mean $\mu$ and the standard deviation $\sigma$.

Standard normal distribution with μ=0 and σ=1.

Expected Values

The expected value of a random variable provides a measure of the "center" of its distribution, akin to an average. For a discrete random variable $X$, the expected value $E(X)$ is calculated as the sum of all possible values of $X$ weighted by their probabilities. For continuous random variables, this involves integrating the product of the variable value and its probability density function over the range of possible values.

The Law of Large Numbers and the Central Limit Theorem

Two pivotal theorems that underpin statistical inference and machine learning are the Law of Large Numbers and the Central Limit Theorem. The Law of Large Numbers states that as the number of trials increases, the sample average of a random variable converges to the expected value. The Central Limit Theorem posits that the sum (or average) of a large number of independent, identically distributed random variables will be approximately normally distributed, regardless of the original distribution. These theorems justify the use of probability in making inferences about populations based on sample data.

Visualization of the Central Limit Theorem and Law of Large Numbers enabling statistical inferences.

Understanding these basics of probability lays a robust foundation for more sophisticated probabilistic models and statistical methods that you will explore in subsequent chapters. As you delve deeper into the world of machine learning, the ability to model, analyze, and infer from uncertain data using probability will be an invaluable skill.