Review of Sample Spaces and Events

To make sense of the probabilistic underpinnings of machine learning algorithms, we need a clear language to describe uncertainty and outcomes. This starts with defining the fundamental concepts of sample spaces and events. Think of these as the basic building blocks for constructing probabilistic models.

The Sample Space: Defining All Possibilities

Before we can talk about the probability of something happening, we first need to precisely define all the possible things that could happen. This complete set of all potential outcomes of a random experiment or process is called the sample space, usually denoted by the Greek letter Omega ($\Omega$).

What constitutes an "outcome" depends entirely on the experiment we're considering:

• Flipping a coin once: The only possible outcomes are heads (H) or tails (T). So, the sample space is $\Omega = \{H, T\}$.
• Rolling a standard six-sided die: The outcomes are the numbers on the faces. $\Omega = \{1, 2, 3, 4, 5, 6\}$.
• Flipping a coin twice: Each outcome is a pair of results. $\Omega = \{(H, H), (H, T), (T, H), (T, T)\}$. We often write this more compactly as $\{HH, HT, TH, TT\}$.
• Measuring the temperature: This is different. Temperature can ideally take any value within a range, not just discrete steps. If we measure temperature in Celsius, the sample space might be represented as an interval of real numbers, for example, $\Omega = \{t \in \mathbb{R} \mid -50 \le t \le 60 \}$. This is an example of a continuous sample space, whereas the previous examples were discrete.

Defining the sample space correctly is the first step in any probability problem. It sets the stage by listing every possibility we need to account for.

Events: Subsets of Interest

Usually, we're not interested in all possible outcomes simultaneously, but rather in whether a specific outcome or a group of outcomes occurred. An event is any subset of the sample space $\Omega$. We often use capital letters like $A$, $B$, $E$, etc., to denote events.

Let's look at events based on our previous sample spaces:

• Rolling a die ($\Omega = \{1, 2, 3, 4, 5, 6\}$):
  • The event "rolling a 6" is the subset $A = \{6\}$.
  • The event "rolling an even number" is the subset $B = \{2, 4, 6\}$.
  • The event "rolling a number greater than 3" is $C = \{4, 5, 6\}$.
• Flipping a coin twice ($\Omega = \{HH, HT, TH, TT\}$):
  • The event "getting exactly one head" is $D = \{HT, TH\}$.
  • The event "getting at least one head" is $E = \{HH, HT, TH\}$.
• Measuring temperature ($\Omega = \{t \in \mathbb{R} \mid -50 \le t \le 60 \}$):
  • The event "temperature is freezing or below" is $F = \{t \in \Omega \mid t \le 0\}$.
  • The event "temperature is between 20°C and 25°C" is $G = \{t \in \Omega \mid 20 \le t \le 25\}$.

Two special events exist for any sample space:

1. The sample space itself ($\Omega$): This is the certain event, because any outcome of the experiment must be within $\Omega$.
2. The empty set ($\emptyset$ or $\{\}$): This is the impossible event, as it contains no outcomes.

Combining Events using Set Operations

Since events are sets, we can use standard set operations to create new events from existing ones. This is essential for understanding relationships between different occurrences.

• Union ($A \cup B$): Represents the outcomes that are in event $A$, or in event $B$, or in both. Read as "$A$ or $B$".
  • Example (Die roll): If $B = \{2, 4, 6\}$ (even) and $C = \{4, 5, 6\}$ (> 3), then $B \cup C = \{2, 4, 5, 6\}$ (even or greater than 3).
• Intersection ($A \cap B$): Represents the outcomes that are common to both event $A$ and event $B$. Read as "$A$ and $B$".
  • Example (Die roll): $B \cap C = \{4, 6\}$ (even and greater than 3).
• Complement ($A^c$ or $A'$): Represents all outcomes in the sample space $\Omega$ that are not in event $A$. Read as "not $A$".
  • Example (Die roll): If $B = \{2, 4, 6\}$ (even), then $B^c = \{1, 3, 5\}$ (not even, i.e., odd).

Two events $A$ and $B$ are called mutually exclusive or disjoint if they have no outcomes in common, meaning their intersection is the empty set ($A \cap B = \emptyset$). For example, the event of rolling an even number and the event of rolling an odd number on a die are mutually exclusive.

A Venn diagram representation using Graphviz for events B (even numbers) and C (numbers > 3) within the sample space Ω of a single die roll. The intersection $B \cap C$ includes outcomes {4, 6}. The complement of B, $B^c$, includes {1, 3, 5}.

Understanding sample spaces and events allows us to precisely define the scenarios we want to analyze. In machine learning, the "experiment" might be observing features of a data point, running an algorithm, or receiving user input. The "outcomes" could be classifications, predicted values, or system states. Events then correspond to specific predictions (e.g., classifying an email as spam) or data characteristics (e.g., a pixel value falling within a certain range). The next step, which we'll explore throughout this course, is assigning probabilities to these events.