How MX Player’s Recommendation Engine Is Built: Architecture & Core Processes

OVERALL ARCHITECTURE

From the architecture diagram, the system is divided into a function part and an algorithm part.

In the function part, the API layer calls the recommendation service to fetch content and enrich details via the DI service. The recommendation service accesses reco/di indexes, a Faiss service for KNN search, algorithm output for recall, and a DNN service for ranking. Simple models are loaded directly in the engine.

The algorithm part collects user logs via a LOG service, streams them to Kinesis, stores them in S3, and trains various models for ranking and data‑mining jobs. Embeddings for each content and user are computed to enable KNN recall. This part is not the focus of the article.

COMMUNICATION WITH APIs

API communication uses Thrift with defined request and response formats. The flow includes time synchronization, request queuing with timestamps, rate limiting, back‑door and validity checks, core processing, and packaging the response.

Time sync for all API and recommendation instances.

Requests are placed in a per‑instance request queue with a timestamp.

Check timestamp to discard outdated requests.

All traffic passes a rate limiter based on pressure‑test thresholds.

Detect back‑door requests.

Validate request integrity.

Enter core process.

Package and return the response to the API.

(If any checkpoint from steps 3‑6 fails, processing jumps directly to packaging.)

CORE PROCESS

The core process performs the main calculations for each interface, with configurable steps. Configuration files are selected per country (e.g., India). Requests are routed to specific interface flow groups based on interface name, platform, and parameters, supporting AB‑test experiments.

After selecting the flow, the system follows the configured component list and order. Key components include:

Recall component: retrieves data from ES, Redis, or Faiss.

Filter component: removes offline or unsuitable content.

Scorer component: computes multi‑dimensional scores using Faiss or DNN services.

Ranker component: orders content based on scorer outputs.

Ruler component: applies rules for post‑ranking adjustments such as diversity.

Packager component: formats the response according to Thrift API definitions.

Cache component: provides recall‑level and sometimes full‑interface caching.

Components access databases or other services via a DataSource Manager. After all components finish, the final response is sent back to the API.

SUMMARY

This article provides a brief introduction to MX Player’s recommendation engine, covering overall architecture, API communication, and internal data processing.