Introducing DeepMatch: An Open‑Source Library for Deep Retrieval Matching Algorithms

DeepMatch is an open‑source project that implements a collection of mainstream deep‑learning based recall‑matching algorithms and enables quick export of user and item vectors for approximate nearest neighbor (ANN) retrieval, making it suitable for fast experiments and learning in recommendation systems.

The project background explains the two‑stage architecture of modern recommendation and advertising systems (recall followed by ranking) and describes the author’s experience building a recommendation system with vector‑based recall, which motivated the creation of DeepMatch.

Installation is straightforward via pip:

pip install -U deepmatch

Documentation and examples are available at https://deepmatch.readthedocs.io/en/latest/ . The repository can be found at https://github.com/shenweichen/DeepMatch .

An example using the YoutubeDNN model demonstrates how to train a recall model, export user and item embeddings, and optionally perform ANN search with Faiss. The full example code is shown below:

import pandas as pd from deepctr.inputs import SparseFeat, VarLenSparseFeat from preprocess import gen_data_set, gen_model_input from sklearn.preprocessing import LabelEncoder from tensorflow.python.keras import backend as K from tensorflow.python.keras.models import Model from deepmatch.models import * from deepmatch.utils import sampledsoftmaxloss # Load data data = pd.read_csv("./movielens_sample.txt") # Define sparse features sparse_features = ["movie_id", "user_id", "gender", "age", "occupation", "zip"] SEQ_LEN = 50 negsample = 0 # Encode features features = ['user_id', 'movie_id', 'gender', 'age', 'occupation', 'zip'] feature_max_idx = {} for feature in features: lbe = LabelEncoder() data[feature] = lbe.fit_transform(data[feature]) + 1 feature_max_idx[feature] = data[feature].max() + 1 # Build user and item profiles user_profile = data[["user_id", "gender", "age", "occupation", "zip"]].drop_duplicates('user_id') item_profile = data[["movie_id"]].drop_duplicates('movie_id') user_profile.set_index("user_id", inplace=True) user_item_list = data.groupby("user_id")["movie_id"].apply(list) # Generate train/test sets train_set, test_set = gen_data_set(data, negsample) train_model_input, train_label = gen_model_input(train_set, user_profile, SEQ_LEN) test_model_input, test_label = gen_model_input(test_set, user_profile, SEQ_LEN) # Feature columns embedding_dim = 16 user_feature_columns = [ SparseFeat('user_id', feature_max_idx['user_id'], embedding_dim), SparseFeat('gender', feature_max_idx['gender'], embedding_dim), SparseFeat('age', feature_max_idx['age'], embedding_dim), SparseFeat('occupation', feature_max_idx['occupation'], embedding_dim), SparseFeat('zip', feature_max_idx['zip'], embedding_dim), VarLenSparseFeat(SparseFeat('hist_movie_id', feature_max_idx['movie_id'], embedding_dim, embedding_name='movie_id'), SEQ_LEN, 'mean', 'hist_len') ] item_feature_columns = [SparseFeat('movie_id', feature_max_idx['movie_id'], embedding_dim)] # Build and train model K.set_learning_phase(True) model = YoutubeDNN(user_feature_columns, item_feature_columns, num_sampled=5, user_dnn_hidden_units=(64, 16)) model.compile(optimizer='adagrad', loss=sampledsoftmaxloss) model.fit(train_model_input, train_label, batch_size=256, epochs=1, verbose=1) # Export embeddings user_embedding_model = Model(inputs=model.user_input, outputs=model.user_embedding) item_embedding_model = Model(inputs=model.item_input, outputs=model.item_embedding) user_embs = user_embedding_model.predict(test_model_input, batch_size=2**12) item_embs = item_embedding_model.predict({"movie_id": item_profile['movie_id'].values, "movie_idx": item_profile['movie_id'].values}, batch_size=2**12) # Optional ANN search with Faiss import faiss, numpy as np index = faiss.IndexFlatIP(embedding_dim) index.add(item_embs) D, I = index.search(user_embs, 50) # Compute recall and hit rate ...

The article also lists contributors, provides contact information, and includes a recruitment notice for positions at Alibaba’s commercial machine intelligence department, inviting interested candidates to apply.

Finally, readers are encouraged to star the GitHub repository, join the DataFunTalk community, and follow additional recommended articles linked at the end of the page.