The Sigmoid Function

In the previous section, we introduced Logistic Regression as a method for binary classification, aiming to predict whether an input belongs to one of two classes (often labeled 0 or 1). But how does an algorithm that looks somewhat like Linear Regression end up predicting a category? Linear Regression produces outputs that can be any real number, which isn't directly useful for deciding between two discrete classes.

We need a way to transform the output of a linear equation, let's call it $z = w \cdot x + b$ (where $w$ represents weights, $x$ the input features, and $b$ the bias), into a value that represents a probability. Probabilities are conveniently bounded between 0 and 1, making them ideal for classification tasks. If the probability is high (close to 1), we can predict class 1; if it's low (close to 0), we predict class 0.

This is where the Sigmoid function, also known as the logistic function, comes into play. It's a mathematical function that takes any real number as input and squashes it into an output between 0 and 1.

The Sigmoid Formula

The sigmoid function, often denoted by the Greek letter sigma $\sigma$, is defined as:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Here, $z$ is the input to the function (which, in Logistic Regression, is the output of the linear part, $w \cdot x + b$), and $e$ is the base of the natural logarithm (approximately 2.718).

Properties and Shape

Let's examine what this function does:

1. Output Range: No matter what value $z$ takes (whether it's -1000, 0, or 500), the output $\sigma(z)$ will always be strictly between 0 and 1. It never actually reaches 0 or 1, but gets very close.
2. Behavior for Large Inputs: As $z$ becomes very large and positive, $e^{-z}$ approaches 0. Therefore, $\sigma(z)$ approaches $\frac{1}{1 + 0} = 1$.
3. Behavior for Large Negative Inputs: As $z$ becomes very large and negative, $e^{-z}$ becomes very large and positive. Therefore, $\sigma(z)$ approaches $\frac{1}{1 + \text{very large number}}$, which gets close to 0.
4. Behavior at Zero: When $z = 0$, $e^{-0} = 1$. So, $\sigma(0) = \frac{1}{1 + 1} = 0.5$.

The function produces a characteristic "S" shape when plotted:

The sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$ smoothly maps any real input $z$ to an output between 0 and 1.

Connecting to Logistic Regression

In Logistic Regression, the model calculates the linear combination $z = w \cdot x + b$ just like in Linear Regression. However, instead of using $z$ directly as the prediction, it feeds $z$ into the sigmoid function:

$$h_\theta(x) = \sigma(z) = \sigma(w \cdot x + b) = \frac{1}{1 + e^{-(w \cdot x + b)}}$$

The output $h_\theta(x)$ (where $\theta$ represents the model's parameters, $w$ and $b$) is now interpreted as the estimated probability that the input $x$ belongs to the positive class (class 1).

For example, if for a given input $x$, the model calculates $z=2$, then the output probability is $\sigma(2) \approx 0.88$. This means the model estimates an 88% chance that this input belongs to class 1. If another input results in $z=-1$, the output probability is $\sigma(-1) \approx 0.27$, indicating a 27% chance of belonging to class 1 (or conversely, a 73% chance of belonging to class 0).

This probabilistic output is fundamental to Logistic Regression. Typically, we set a decision threshold (often 0.5) to convert this probability into a definite class prediction. If $h_\theta(x) \ge 0.5$, we predict class 1; otherwise, we predict class 0. The point where $z=0$ and $\sigma(z)=0.5$ often corresponds to the boundary separating the predicted classes, which we will explore next when discussing decision boundaries.