Constructing a Simple Autoencoder Model

With our Python environment ready and a grasp of the dataset (like MNIST, which we'll assume for our examples), we can now define the structure of our simple autoencoder. We'll use Keras, a high-level API within TensorFlow, which makes building neural networks quite accessible.

Building Blocks: Layers in Keras

An autoencoder, at its heart, is a neural network. We'll construct it using layers. For this basic model, we'll primarily use Dense layers. A Dense layer is a fully connected layer, meaning each neuron in it receives input from all neurons in the previous layer.

Our autoencoder will have two main parts:

1. The Encoder: This part takes the input data and compresses it down to a smaller representation, often called the "bottleneck" or "latent space."
2. The Decoder: This part takes the compressed representation from the bottleneck and tries to reconstruct the original input data from it.

We'll use the Keras Sequential model, which is ideal for creating models as a linear stack of layers.

Defining the Encoder

The encoder's job is to reduce the dimensionality of the input. If our input data is, for example, a flattened MNIST image with $28 \times 28 = 784$ pixels, the encoder will map these 784 dimensions to a much smaller number.

1. Input Layer: The first layer in our encoder needs to know the shape of our input data. For flattened MNIST images, this would be 784 features. We specify this using the input_shape argument in the first Dense layer.
2. Hidden Layers (Encoder): We can add one or more Dense layers that progressively reduce the number of neurons. For example, from 784 inputs, we might go to 128 neurons.
3. Bottleneck Layer: This is the final layer of the encoder and has the smallest number of neurons. This layer produces the compressed representation. Let's say we compress it down to 64 neurons. The size of this layer, encoding_dim, is a critical parameter.
4. Activation Functions: For the hidden layers in the encoder, including the bottleneck, we'll use the ReLU (Rectified Linear Unit) activation function. ReLU is a common choice because it helps with training deep networks and is computationally efficient. It simply outputs the input if it's positive, and zero otherwise: $f(x) = \max(0, x)$.

Defining the Decoder

The decoder's task is the reverse of the encoder. It takes the compressed representation from the bottleneck and expands it back to the original input's dimensions.

1. Hidden Layers (Decoder): These layers will increase the number of neurons, mirroring the encoder but in reverse. For instance, from the 64 neurons in the bottleneck, we might expand to 128 neurons.
2. Output Layer: The final layer of the decoder must have the same number of neurons as the original input data (e.g., 784 for MNIST). This layer produces the reconstructed data.
3. Activation Function (Output): For the output layer, a common choice is the Sigmoid activation function, especially if the input data (like pixel values) was normalized to be between 0 and 1. The Sigmoid function, $f(x) = \frac{1}{1 + e^{-x}}$, squashes its output into the range (0, 1), making it suitable for reconstructing normalized data.

Putting It Together with Keras Code

Let's translate this into Keras code. First, ensure you have TensorFlow installed and imported.

from tensorflow import keras
from tensorflow.keras import layers

# Define the dimensionality of the input.
# For MNIST, each image is 28x28 pixels, so flattened it's 784.
input_dim = 784

# Define the size of the encoded representation (the bottleneck).
# This is a hyperparameter you can tune. Let's choose 64.
encoding_dim = 64

# Define the autoencoder model using the Keras Sequential API
autoencoder = keras.Sequential([
    # Encoder part
    layers.Dense(128, activation='relu', input_shape=(input_dim,)), # Input layer connected to first hidden layer
    layers.Dense(encoding_dim, activation='relu'),                  # Bottleneck layer

    # Decoder part
    layers.Dense(128, activation='relu'),                           # First hidden layer of decoder
    layers.Dense(input_dim, activation='sigmoid')                   # Output layer, reconstructing the input
])

# Display the model's architecture
autoencoder.summary()

When you run autoencoder.summary(), Keras will print a table that looks something like this (the exact number of parameters might vary slightly based on TensorFlow/Keras versions but the structure will be consistent):

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 128)               100480    
                                                                 
 dense_1 (Dense)             (None, 64)                8256      
                                                                 
 dense_2 (Dense)             (None, 128)               8320      
                                                                 
 dense_3 (Dense)             (None, 784)               101136    
                                                                 
=================================================================
Total params: 218,192
Trainable params: 218,192
Non-trainable params: 0
_________________________________________________________________

Understanding the Summary:

• Layer (type): Shows the name and type of each layer. Keras automatically names them if you don't.
• Output Shape: (None, 128) means this layer outputs a tensor where the first dimension (None) is the batch size (which can vary), and the second dimension is the number of neurons (128 in this case).
• Param #: This is the number of trainable parameters (weights and biases) in that layer. For example, the first Dense layer connects 784 inputs to 128 neurons. This involves $784 \times 128$ weights plus 128 biases, totaling $100352 + 128 = 100480$ parameters.

Visualizing the Model Architecture

A diagram can help visualize the flow of data through our autoencoder:

This diagram shows the data flowing from the input layer, through the encoder (compressing to 64 units at the bottleneck), and then through the decoder to reconstruct the original 784 units at the output.

In this structure:

• We use Dense layers because they are versatile and a good starting point for many neural network tasks, including basic autoencoders.
• ReLU (activation='relu') is used in the hidden layers. It's a popular choice that helps models learn effectively.
• Sigmoid (activation='sigmoid') is used in the output layer. If your input pixel values are scaled between 0 and 1 (a common preprocessing step for images), sigmoid ensures the output values are also in this range, which is helpful for comparing the reconstruction to the original.

With the model's architecture defined, the next step is to configure it for training. This involves choosing a loss function (to measure how well the autoencoder is doing) and an optimizer (to tell the model how to get better). We'll cover that in the next section.