How User‑Based Collaborative Filtering Powers Modern Recommendation Systems

Recommendation algorithms were first proposed in 1992 but only gained popularity with the explosion of internet data, enabling personalized suggestions when users themselves do not know what they want.

Basic Conditions of Recommendation

Recommend based on users with similar preferences.

Recommend items similar to those the user likes.

Recommend using keywords (essentially a search).

Combine the above conditions.

User‑Based Collaborative Filtering

This algorithm treats the user as the primary entity, emphasizing social relationships: it recommends items liked by users with similar tastes, contrasting with item‑based methods that focus on item similarity.

Similarity between users is computed using classic metrics such as Jaccard (intersection over union), cosine similarity, or Euclidean distance, with the choice depending on the data characteristics.

Finding the K Nearest Neighbors

For a target user, we compare all other users and select the K most similar ones (the "good friends"). To reduce computation on large datasets, we first build an item‑to‑user reverse index so that only users sharing items with the target are considered.

Scoring Recommendations

Each neighbor contributes to the recommendation score of items they like, weighted by their similarity to the target user. For example, if neighbor A has similarity 0.25 and neighbor B 0.80, the scores for items they liked are calculated as:

Item X: 1 × 0.25 = 0.25

Item Y: 1 × 0.80 = 0.80

Item Z: 1 × 0.80 + 1 × 0.25 = 1.05

Items are then ranked by these scores, and the highest‑scoring items are recommended.

Algorithm Summary

Compute similarity between the target user and other users, using the reverse index to ignore unrelated users.

Select the K most similar neighbors.

Aggregate the items liked by these neighbors, weighting each by the neighbor’s similarity.

Rank items by their aggregated scores and present the top recommendations.

Practical Issues

Popular items may dominate recommendations, and overly generic items (e.g., dictionaries) provide little value; such "dirty data" should be filtered or down‑weighted during preprocessing.

Real‑World Example with MovieLens

Using the MovieLens dataset, we treat ratings above 3 (or above a user’s average rating) as positive feedback. The following Python‑style pseudocode illustrates the workflow:

# Read file data
test_contents = readFile(file_name)
# Convert to list of [user_id, movie_id, rating]
test_rates = getRatingInformation(test_contents)
# Build dictionaries: user->[(movie, rating)...] and movie->[user...]
test_dic, test_item_to_user = createUserRankDic(test_rates)
# Find K nearest neighbors
neighbors = calcNearestNeighbor(userid, test_dic, test_item_to_user)[:k]
# Aggregate recommendation scores
recommend_dic = {}
for neighbor in neighbors:
    neighbor_user_id = neighbor[1]
    movies = test_dic[neighbor_user_id]
    for movie in movies:
        if movie[0] not in recommend_dic:
            recommend_dic[movie[0]] = neighbor[0]
        else:
            recommend_dic[movie[0]] += neighbor[0]
# Build sorted recommendation list
recommend_list = []
for key in recommend_dic:
    recommend_list.append([recommend_dic[key], key])
recommend_list.sort(reverse=True)

Running this pipeline for a sample user yields recommendations such as "Contact (1997)", "Scream (1996)", "Titanic (1997)", etc. Popular movies like "Titanic" or "Star Wars" often appear for users who have not yet watched them, illustrating the need to handle popularity bias.