From Neighborhoods to Latent Factor Models

Neighborhood-based collaborative filtering is an intuitive method that generates recommendations by finding users or items with similar rating histories. This approach, often considered memory-based, works directly with the user-item interaction matrix, identifying "neighbors" to predict a user's preference. While such methods can be effective, their performance can degrade as datasets become larger and sparser. For instance, calculating similarities across millions of users or items is computationally intensive. Furthermore, if two items have never been co-rated, their similarity cannot be determined, a common issue in sparse datasets.

To overcome these challenges, we now shift our attention to model-based collaborative filtering. Instead of relying on the entire dataset at prediction time, these methods use the interaction data to train a more compact model that learns the underlying patterns of user taste. This trained model can then make predictions efficiently without needing to scan through all user-item interactions again.

The most prominent family of model-based techniques revolves around discovering latent factors. These are hidden features that help explain the observed ratings. For a movie dataset, these factors might represent genres like "sci-fi" or "comedy," the presence of a certain director, or more abstract attributes like "coming-of-age story" or "high-octane action." The important part is that we do not need to specify these factors beforehand; the model learns them automatically from the patterns in the ratings data.

The core idea is to represent both users and items in a shared, lower-dimensional latent space.

Each user is described by a vector that measures their affinity for each latent factor. For example, a user's vector might indicate a strong preference for "sci-fi" and a low interest in "comedy."
Each item is also described by a vector that measures how much it embodies each of those same factors. A movie's vector would show it is high on "sci-fi" but low on "comedy."

A recommendation is made by comparing a user's vector to an item's vector in this latent space. If the vectors are well-aligned, meaning the user likes the factors that the item possesses, the model predicts a high rating. This prediction is typically calculated as the dot product of the two vectors. This approach allows the model to generalize. It can recommend a sci-fi movie to a user who loves sci-fi, even if that user has never rated a movie from the same director or with the same actors before.

Neighborhood-based methods rely on direct, observable connections (left), whereas latent factor models map users and items to a shared feature space to infer preferences (right).

This brings us to matrix factorization, the primary technique for revealing these latent factors. As outlined in the chapter introduction, matrix factorization decomposes the large, sparse user-item interaction matrix ( $R$ ) into two smaller, dense matrices: a user-factor matrix ( $P$ ) and an item-factor matrix ( $Q$ ). The dot product of a user's vector from $P$ and an item's vector from $Q$ gives us the predicted rating $\hat{r}_{ui}$ .

\hat{r}_{ui} = p_u \cdot q_i

By learning these factor matrices, we create a powerful and compact representation of user preferences and item attributes. This model-based approach offers several advantages over its neighborhood-based counterparts:

Better Handling of Sparsity: It can estimate ratings for all user-item pairs, not just those with overlapping interaction data.
Improved Scalability: Once the model is trained, making predictions is extremely fast, as it only requires a dot product calculation rather than iterating over neighbors.
Compact Representation: Instead of storing a massive and sparse user-item matrix, we only need the two smaller factor matrices, which is far more memory-efficient.

In the sections that follow, we will explore the mechanics of matrix factorization, starting with one of its most well-known algorithms, Singular Value Decomposition (SVD).

Was this section helpful?

References

Matrix Factorization Techniques for Recommender Systems, Yehuda Koren, Robert M. Bell, Chris Volinsky, 2009 Computer, Vol. 42(8) (IEEE) DOI: 10.1109/MC.2009.263 - This seminal paper introduces and details various matrix factorization techniques, including SVD-based methods, which became foundational for modern recommendation systems.
Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira, 2015 (Springer) DOI: 10.1007/978-1-4899-7637-6 - A comprehensive textbook providing in-depth coverage of all major recommender system techniques, with dedicated sections on collaborative filtering and matrix factorization.
Application of Dimensionality Reduction in Recommender Systems - A Case Study, Badrul M. Sarwar, George Karypis, Joseph A. Konstan, John T. Riedl, 2000 Proceedings of the WebKDD 2000 Workshop at the ACM-SIGKDD Conference on Knowledge Discovery in Databases (ACM) DOI: 10.1145/347090.347101 - One of the earliest influential works demonstrating the use of Singular Value Decomposition (SVD) for collaborative filtering to address sparsity and scalability challenges.