Ranking Metrics: Precision and Recall at K

Evaluating recommendation system performance often involves assessing different aspects. While some evaluation approaches measure how accurately a model predicts specific ratings, using metrics like RMSE and MAE, most recommendation applications prioritize a different objective: creating a ranked list of items where the best ones appear at the top. The user typically doesn't care if the system predicted they would rate a movie 4.2 stars or 4.3 stars; their primary concern is whether a desirable movie appeared in their "Top 10 For You" list and if they actually liked it.

This is where ranking metrics come into play. They shift the focus from prediction accuracy to the quality of the ordered list of recommendations. The two most fundamental ranking metrics are Precision and Recall, which help us answer two simple but important questions:

1. Out of the items we recommended, how many were actually relevant? (Precision)
2. Out of all the items the user would have found relevant, how many did we successfully recommend? (Recall)

These metrics are typically evaluated at a specific cutoff point, $k$, leading to the terms Precision at $k$ (P@k) and Recall at $k$ (R@k). The value of $k$ is usually tied to the application's user interface, such as the top 5 items shown in a mobile app's banner or the top 20 items in a weekly email.

Precision at K

Precision at $k$ measures the proportion of recommended items in the top-$k$ set that are actually relevant. It's a measure of exactness or quality. A high precision means that the recommender is good at presenting items the user will like.

The formula for Precision@k is straightforward:

$$\text{Precision@k} = \frac{\text{|Relevant Items Recommended in Top k|}}{k}$$

Let's walk through an example. Imagine our system generates a top-10 list of movie recommendations for a user. We check this list against a held-out test set of movies that we know the user has watched and liked (our "relevant" items).

• Recommended Top 10: [Movie A, Movie B, Movie C, Movie D, Movie E, Movie F, Movie G, Movie H, Movie I, Movie J]
• User's Relevant Items in Test Set: [Movie B, Movie E, Movie I, Movie K, Movie M, Movie P]

To calculate P@10, we see which of the recommended movies are in the user's relevant set. In this case, Movie B, Movie E, and Movie I are the overlapping items. There are 3 such items.

So, the Precision@10 is:

$$\text{P@10} = \frac{3}{10} = 0.3$$

This means 30% of our top-10 recommendations were relevant to the user.

Recall at K

Recall at $k$ measures the proportion of all relevant items that are successfully captured in the top-$k$ recommendations. It's a measure of completeness. A high recall means the system is good at finding most of the items the user would like.

The formula for Recall@k is:

$$\text{Recall@k} = \frac{\text{|Relevant Items Recommended in Top k|}}{\text{|Total Relevant Items for the User|}}$$

Using the same example as before:

• Relevant items recommended in Top 10: Movie B, Movie E, Movie I (Count = 3)
• Total relevant items for the user in the test set: Movie B, Movie E, Movie I, Movie K, Movie M, Movie P (Count = 6)

The Recall@10 is calculated as:

$$\text{R@10} = \frac{3}{6} = 0.5$$

This result means our top-10 list managed to find 50% of the total items the user would have found relevant.

The Precision and Recall Trade-off

You might notice an inherent tension between precision and recall. If you increase $k$ by recommending more items, you increase your chances of including more relevant items, which generally raises recall. However, by lengthening the list, you also increase the risk of including irrelevant items, which can lower your precision.

Conversely, if you make your recommendation list very short (a small $k$) and only include items you are highly confident about, you might achieve high precision. But you will likely miss many other relevant items, resulting in low recall. This inverse relationship is a classic trade-off in information retrieval and machine learning.

As the number of recommendations ($k$) grows, recall tends to increase because more relevant items are likely to be included. At the same time, precision often decreases as the list becomes diluted with less relevant items.

Choosing an Appropriate Value for K

The choice of $k$ is not just a statistical decision; it's a product design decision. You should choose a value for $k$ that reflects how recommendations are presented to the user.

• If your website has a "Top 5 Recommended For You" section on the homepage, evaluating with P@5 and R@5 makes the most sense.
• If you send a promotional email with 20 product suggestions, then P@20 and R@20 would be more appropriate metrics to track.

By aligning your offline evaluation metric with the actual user experience, you get a much more realistic assessment of your model's performance.

Defining Relevance and Limitations

Before you can calculate precision or recall, you must first define what makes an item "relevant."

• Explicit Data: With ratings data, a common approach is to set a threshold. For example, any movie a user rated 4 or 5 stars is considered relevant, while anything rated lower is not.
• Implicit Data: With implicit interactions like clicks, views, or purchases, any recorded positive interaction is often treated as a signal of relevance.

It is also important to recognize that P@k and R@k have a limitation: they are insensitive to the ordering of items within the top-$k$ list. For these metrics, a relevant item at position 1 has the exact same value as a relevant item at position $k$. They simply treat the top-$k$ recommendations as an unordered set. In many applications, however, getting the top item right is much more valuable than getting the tenth item right. For that, we need more advanced, rank-aware metrics like Mean Average Precision (MAP) and NDCG, which we will cover next.