Building a Real‑Time Recommendation Engine with Flink: A Complete Example Project

End‑to‑End Recommendation Pipeline

A recommendation system is a long‑term project; the chain from behavior collection to real‑time computation to online service must stay intact, otherwise recommendations become incorrect and users simply swipe away.

Data Flow Overview

User actions flow as follows:

user behavior → Kafka → Flink (real‑time compute) → Redis/HBase/ES → Online recommendation API → Front‑end

Feedback from the front‑end loops back into the pipeline.

Example behavior event (JSON):

{
  "userId": "user_001",
  "productId": 42,
  "action": "purchase",
  "scene": "home-feed",
  "timestamp": 1783899824178
}

The action field has five possible values: browse, click, collect, purchase, dislike. All Flink jobs, profiling, and recall logic share this semantic set.

Kafka: Event Bus

All behavior events are written to the user-behavior topic. Downstream consumers read only the fields they need: hot‑list tasks focus on product dimension, profiling tasks on user dimension, and log aggregation on overall volume. Consumer groups are isolated, preventing interference.

Kafka is used as a bus instead of direct RPC because behavior data naturally supports multiple consumers—hot‑list, profiling, and BI reports can all read the same stream without coupling or added latency.

Flink Real‑Time Computation

Five ProcessFunctions implement the core logic:

Hot‑list calculation: Sliding window weighted by action type (browse 1, click 3, collect 5, purchase 10). Top N results are written to a Redis Sorted Set. Default window: 1 hour, slide: 5 minutes.

User profiling: Each user maintains a MapState where keys are tags (color, style, category, price range) and values are cumulative weights. Every incoming event adds the corresponding weight; a timer flushes the state to HBase every 30 seconds.

Item profiling: Same logic as user profiling but with item‑side dimensions (user groups that like the item, conversion rate, popularity trend).

Behavior history: Stores the most recent N interaction records for collaborative filtering and duplicate‑view removal. Stored in HBase with row key userId_productId, TTL 7 days.

Log aggregation: Minute‑level action counts are written to Elasticsearch for front‑end dashboards.

All Flink task outputs are persisted to storage rather than pushed directly to the online service, decoupling the two layers. If Flink restarts, only a few minutes of profiling delay occurs, while the recommendation API remains available.

Storage: Redis + HBase + Elasticsearch

Redis: Holds hot data—real‑time hot‑list (Sorted Set, TTL 2 h), recommendation cache per user (TTL 5 min), short‑term counters. Read latency < 1 ms.

HBase: Stores wide‑table data: user profiles (row key = userId), item profiles (row key = productId), behavior history. Point lookups suit millions of users and items.

Elasticsearch: Stores logs and analytical data for trend charts, anomaly detection, and also serves as the product search engine (full‑text on title, tags, category).

The storage split aims to keep the online recommendation API’s P99 latency under 50 ms: Redis and HBase provide millisecond reads, while Elasticsearch is used only for search, not the main recommendation path.

Online Recommendation Engine

The request processing consists of five steps:

Recall: Four parallel sources—hot‑list from Redis (Top 50), collaborative filtering from HBase (user history), profile matching using user tags against item tags, and vector ANN retrieval (cosine similarity simulated; production may use Milvus or Faiss).

Fusion: De‑duplicate the four result sets, filter already‑viewed items, and truncate each source to at most 20 items, keeping the fused list under 80 items.

Feature concatenation: Attach user features (profile tags, recent actions), item features (popularity, conversion rate, price range), and context features (scene, time, device) to each candidate. The resulting vector feeds the ranking model.

Scoring (ranking): Example uses a weighted sum:

score = 0.3 × hotness + 0.4 × profileMatch + 0.3 × collaborative

. In production this would be replaced by a GBDT or deep model without changing the API.

Re‑ranking (business rules): Apply constraints such as inventory check, category dispersion (no three consecutive items of the same category), blacklist, and content safety filters. Finally return the top N items.

The API also returns a pipeline array containing the input/output counts and latency of each step, enabling quick diagnosis of slow stages.

Front‑End Console (Ant Design)

The system provides five pages:

Dashboard: Real‑time behavior trends, hot‑list Top 10, event stream, component status; charts built with ECharts and data pushed via WebSocket.

Recommendation Scheduler: Select user, scene, and strategy (hot / cf / profile / hybrid); trigger recommendation request; view product cards with score and reason; simulate clicks or collects that feed back into the pipeline; a panel shows per‑step item counts and latency.

Product Management: List, search, and view product details, including its profile tags and hotness trend.

User Profile: Input a userId to display a radar chart of tag preferences, recent behavior timeline, and recommendation history.

System Monitoring: Shows connection status and mode for Kafka, Flink, Redis, HBase, and Elasticsearch; displays “memory downgrade” in local mode and real connections in full mode.

What Remains for Production

Feature platform: Current features are scattered across Flink jobs and backend code. A production system needs a unified feature registry, lineage tracking, and consistency checks between training and serving; otherwise offline AUC may drop from 0.75 to 0.68 without a clear cause.

Experiment platform: Needs A/B traffic splitting, sample size calculation, significance testing, and mutual exclusion. The example only includes an experimentId field without actual split logic.

Model service: Replacing the weighted scoring with a deep model requires model registration, version management, A/B comparison, and automatic rollback; integration options include TensorFlow Serving or ONNX Runtime.

Vector retrieval: The cosine‑similarity loop works for small data but does not scale; production should adopt Milvus or Elasticsearch dense_vector with HNSW indexing.

Data quality monitoring: Monitor event latency, missing fields, traffic anomalies, and data drift; garbage‑in‑garbage‑out will render even the best model useless.

These components are not optional; without them a real system would quickly break. The example project leaves interfaces and data structures in place so that extending the pipeline does not require redesigning the core flow.

Project Structure

The repository is organized as follows:

recommendation-system/
  flink-jobs/          # Flink streaming tasks
  backend/             # Spring Boot recommendation API
  frontend/            # React + Ant Design console
  data-simulator/      # Python behavior event generator
  infra/               # Kafka/HBase/ES initialization scripts
  docker-compose.yml

Core APIs:

GET  /api/recommend?userId=user_001&topN=8&strategy=hybrid
POST /api/recommend/behavior
GET  /api/dashboard
GET  /api/engine/overview
GET  /api/recommend/user/{userId}/profile

Running locally does not require external middleware; simply execute mvn spring-boot:run and npm run dev to start the full end‑to‑end flow.