Fundamentals of Neural Networks

While various machine learning models provide powerful tools for many tasks, neural networks, inspired by the structure of the human brain, have enabled significant progress on more complex problems, particularly in areas like image and speech recognition. The foundational elements of neural networks are examined here, preparing you to implement these models using Julia.

The Neuron: A Basic Computational Unit

At its core, a neural network is composed of interconnected processing units called neurons (or nodes). A single neuron takes multiple input values, performs a calculation, and produces an output. Think of it as a small computational function.

Each input $x_i$ to a neuron is associated with a weight $w_i$. The neuron sums up all these weighted inputs and adds a bias term $b$. This sum, often called the weighted sum or affine transformation, is then passed through an activation function $f$. The output $y$ of the neuron can be expressed as:

$y = f\left(\sum_{i} (w_i x_i) + b\right)$

The weights $w_i$ determine the importance of each input signal, while the bias $b$ acts like an offset, allowing the neuron to activate even if all inputs are zero, or shifting the activation function's effective range. These weights and biases are the parameters that the network "learns" during the training process.

A single neuron computes a weighted sum of its inputs, adds a bias, and then applies an activation function to produce an output.

Layers: Organizing Neurons

Neurons are typically organized into layers:

1. Input Layer: This layer receives the raw data (features) for your machine learning task. The number of neurons in the input layer usually corresponds to the number of features in your dataset. No computation is performed in this layer; it simply passes the data to the first hidden layer.
2. Hidden Layers: These layers lie between the input and output layers. Each neuron in a hidden layer applies a transformation (weighted sum + activation) to the outputs from the preceding layer. The presence of one or more hidden layers allows the network to learn more complex representations of the data. The "deep" in "deep learning" refers to having multiple hidden layers.
3. Output Layer: This layer produces the final result of the network. The number of neurons and the activation function used in the output layer depend on the type of problem you are solving:
   • For regression tasks (predicting a continuous value), the output layer typically has one neuron with a linear activation function (or no activation function).
   • For binary classification (predicting one of two classes), it often has one neuron with a sigmoid activation function.
   • For multi-class classification (predicting one of several classes), it usually has $N$ neurons (where $N$ is the number of classes) and a softmax activation function.

Information flows from the input layer, through one or more hidden layers, to the output layer. Each connection between neurons has an associated weight.

Activation Functions: Introducing Non-Linearity

Activation functions are a critical part of a neural network. If neurons only performed weighted sums, the entire network, no matter how many layers it had, would behave like a single linear model. Activation functions introduce non-linearities, enabling the network to learn complex patterns and relationships in the data that extend past simple linear combinations.

Here are a few common activation functions:

• Sigmoid: As mentioned in the chapter introduction, the sigmoid function is defined as: $\sigma(x) = \frac{1}{1 + e^{-x}}$ It squashes its input into a range between 0 and 1. This makes it suitable for output neurons in binary classification problems, where the output can be interpreted as a probability. However, sigmoid functions can suffer from the "vanishing gradient" problem during training, especially in deep networks, which can slow down learning.

• Hyperbolic Tangent (tanh): $\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$ The tanh function is similar to sigmoid but squashes values to a range between -1 and 1. It is often preferred over sigmoid for hidden layers as its output is zero-centered. It also suffers from the vanishing gradient problem.

• Rectified Linear Unit (ReLU): $\text{ReLU}(x) = \max(0, x)$ ReLU is currently one of the most popular activation functions for hidden layers. It outputs the input directly if it is positive, and zero otherwise. It is computationally efficient and helps mitigate the vanishing gradient problem for positive inputs. Variations like Leaky ReLU or Parametric ReLU (PReLU) exist to address the "dying ReLU" problem (where neurons can become inactive if their input is always negative).

• Softmax: For multi-class classification problems, the softmax function is commonly used in the output layer. It takes a vector of arbitrary real-valued scores (logits) and converts them into a probability distribution over $K$ classes, where each probability is between 0 and 1, and all probabilities sum to 1. For an input vector $z = [z_1, z_2, \ldots, z_K]$, the softmax output for the $j$-th element is: $\text{softmax}(z)_j = \frac{e^{z_j}}{\sum_{k=1}^{K} e^{z_k}}$

Comparison of Sigmoid and ReLU activation functions. Sigmoid maps inputs to (0,1), suitable for probabilities. ReLU outputs $\max(0,x)$, promoting sparsity and alleviating vanishing gradients for positive inputs.

Information Flow: The Feedforward Pass

In a standard feedforward neural network, information flows in one direction: from the input layer, through any hidden layers, to the output layer. This process is called a feedforward pass or forward propagation.

During a feedforward pass:

1. The input data is fed into the input layer.
2. Each neuron in the subsequent layer computes its weighted sum of inputs from the previous layer, adds its bias, and applies its activation function.
3. This process repeats layer by layer until the output layer produces the network's prediction.

For example, the output $\hat{y}$ (the prediction) is generated by processing the input $X$ through the network's functions and learned parameters (weights $W$ and biases $b$).

Common Network Architectures

While the simple feedforward structure described above is fundamental, various specialized architectures have been developed for different types of data and tasks:

• Feedforward Neural Networks (FFNNs) / Multi-Layer Perceptrons (MLPs): These are the networks we've primarily discussed so far. They consist of an input layer, one or more hidden layers, and an output layer, with connections typically flowing only in the forward direction. They are versatile and used for many tabular data classification and regression tasks. You'll start by building these with Flux.jl.
• Convolutional Neural Networks (CNNs or ConvNets): These are particularly effective for grid-like data, such as images. CNNs use special layers (convolutional layers, pooling layers) to automatically and adaptively learn spatial hierarchies of features from images.
• Recurrent Neural Networks (RNNs): Designed for sequential data where order matters, like time series or natural language. RNNs have connections that form directed cycles, allowing them to maintain an internal state or "memory" of past inputs. Variations like LSTMs (Long Short-Term Memory) and GRUs (Gated Recurrent Units) address some challenges in training standard RNNs.

This chapter will focus on feedforward neural networks, providing the basis for understanding more complex architectures later.

How Neural Networks "Learn"

The "learning" in neural networks refers to the process of finding the optimal set of weights and biases that allow the network to map input data to correct outputs. This is typically achieved by:

1. Defining a Loss Function: A loss function (or cost function) measures how far the network's predictions ($\hat{y}$) are from the actual target values ($y$). For example, the Mean Squared Error (MSE), $L = \frac{1}{N}\sum_{i=1}^{N}(y_i - \hat{y}_i)^2$, is often used for regression tasks.
2. Optimization: The goal is to minimize this loss function. This is usually done using an optimization algorithm, most commonly a form of gradient descent.
3. Backpropagation: To use gradient descent, we need to calculate the gradient of the loss function with respect to each weight and bias in the network. Backpropagation is an efficient algorithm for computing these gradients, starting from the output layer and working backward through the network.

You'll explore loss functions, optimizers, and the training process, including automatic differentiation with Zygote.jl, in the subsequent sections of this chapter. For now, understand that these fundamental components, neurons, layers, activation functions, and a forward flow of information, are the building blocks upon which the learning process operates.