Foundations of Representation Learning

Before constructing advanced autoencoder models, it's necessary to understand the foundations of representation learning. High-dimensional data often requires methods to find more compact or meaningful ways to express it. This chapter begins by reviewing core unsupervised learning concepts. We then examine common linear dimensionality reduction techniques, such as Principal Component Analysis (PCA), and discuss their limitations when dealing with complex data structures. This motivates the need for non-linear approaches.

We will introduce manifold learning techniques like $t$-SNE ($t$-Distributed Stochastic Neighbor Embedding) and UMAP (Uniform Manifold Approximation and Projection) as examples of non-linear visualization and reduction methods. The chapter also provides a primer on the information bottleneck theory and refreshes essential mathematical concepts from probability, information theory, and optimization that support the models discussed later. Completing this chapter provides the necessary background for building and understanding the autoencoder architectures presented subsequently.