In the previous chapter, we looked at ways to describe and summarize data we already have. Now, we shift our focus to dealing with uncertainty. Many situations in machine learning, like predicting future outcomes or understanding model behavior, involve processes where the result isn't known in advance. Probability provides the tools to think about and quantify this uncertainty. Let's start with the basic building blocks.

Experiments and Outcomes

In probability, an experiment (or trial) refers to any process or action that has an observable result, but where the specific result cannot be predicted with certainty beforehand. Think of it as a procedure that can be repeated and has a well-defined set of possible results.

Some simple examples include:

Flipping a coin once.
Rolling a standard six-sided die.
Measuring the height of a randomly selected student.
Observing whether a customer clicks on an online ad.

Each potential result of an experiment is called an outcome.

For flipping a coin, the possible outcomes are "Heads" or "Tails".
For rolling a die, the possible outcomes are the numbers 1, 2, 3, 4, 5, or 6.
For measuring height, an outcome might be a specific value like 172.5 cm.
For the ad click, the outcomes are "Click" or "No Click".

Outcomes are the most basic, individual results we can observe from an experiment.

Sample Spaces

While knowing the possible outcomes is useful, we often need a complete picture of all possibilities. The sample space of an experiment is the set of all possible distinct outcomes. It represents the entire universe of what could happen in a single trial of the experiment. We usually denote the sample space with the capital letter $S$ .

Let's define the sample spaces for our examples:

Coin Flip: The outcomes are Heads (H) and Tails (T). The sample space is $S = \{H, T\}$ .
Die Roll: The outcomes are the numbers on the faces. The sample space is $S = \{1, 2, 3, 4, 5, 6\}$ .
Ad Click: The outcomes are Click (C) and No Click (NC). The sample space is $S = \{C, NC\}$ .

The sample space must be exhaustive, meaning it includes every possible outcome. It must also consist of outcomes that are mutually exclusive, meaning that only one outcome can occur in a single trial of the experiment (e.g., a single coin flip cannot be both Heads and Tails). Defining the sample space correctly is a fundamental first step in solving any probability problem.

Events

Often, we are interested not just in a single outcome, but in a specific collection or subset of outcomes. An event is any subset of the sample space. It represents a particular result or group of results that we might care about. Events are usually denoted by capital letters like $A$ , $B$ , $E$ , etc.

Consider rolling a standard six-sided die, where the sample space is $S = \{1, 2, 3, 4, 5, 6\}$ . Here are some possible events:

Event A: Rolling a 3. This event corresponds to a single outcome. As a subset of the sample space, we write it as $A = \{3\}$ .
Event B: Rolling an even number. This event includes multiple outcomes: 2, 4, and 6. As a subset, $B = \{2, 4, 6\}$ .
Event C: Rolling a number greater than 4. This event includes the outcomes 5 and 6. As a subset, $C = \{4, 5, 6\}$ .
Event D: Rolling a 7. Since 7 is not a possible outcome in our sample space $S$ , this event corresponds to the empty set: $D = \{\}$ or $D = \emptyset$ . This is an impossible event for this experiment.
Event E: Rolling a number less than 10. All possible outcomes (1 through 6) satisfy this condition. So, this event is the entire sample space: $E = \{1, 2, 3, 4, 5, 6\} = S$ . This is a certain event.

Understanding these terms is essential:

An experiment is the process.
An outcome is a single possible result.
The sample space ( $S$ ) is the set of all possible outcomes.
An event ( $A, B, ...$ ) is a subset of the sample space, representing a result or collection of results we are interested in.

In machine learning, we often deal with data points. You can think of observing a single data point (like a customer's purchase amount or whether an email is spam) as an outcome of an experiment. The sample space represents all possible observations, and an event might correspond to observing a data point with specific characteristics (e.g., purchase amount over $100, or email classified as spam).

With these foundational concepts of sample spaces and events defined, we can now move on to actually calculating the probabilities associated with these events.