Build a Scalable Spring Cloud Recommendation System: Architecture, Code, and Fault Tolerance

Introduction

Douyin's recommendation engine relies on a complex microservice architecture to handle high concurrency and massive user data. This guide builds a simplified version of that system with Spring Cloud, demonstrating how to design, implement, and test the core services.

Business Analysis

User Behavior Data

Record user actions : capture video watches, likes, shares, etc.

Manage user profile : generate an interest profile from historical actions.

Video Resource Management

Store video metadata : ID, title, tags, upload time.

Provide video categorization for recommendation.

Personalized Recommendation

Match user profile with video tags to generate a recommendation list.

Return the list to the client.

Real‑time Processing of User Behavior

Process events as they occur and update the user profile.

Asynchronous handling via message queues to avoid blocking the user experience.

The core functional requirements are:

Record user behavior.

Manage video resources.

Generate personalized recommendations.

Process data in real time.

Architecture Design

The system is split into four main services registered with Eureka:

Eureka Server

User Service

Video Service

Recommendation Service

Data‑Processing Service

An architecture diagram (simplified) is shown below:

Specific Implementation

Project Setup

Use Maven archetype maven-archetype-quickstart to create the parent project and four module sub‑projects.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://mmbiz.qpic.cn/mmbiz_jpg/sXiaukvjR0RCmCvEYDibYAkuUoia9ErawMUTFrUEqxsM2YUPlvb80nk1icg420iau2f0vn1rSXGPkMhticnz3VfMQ7vA/640" ...>
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.itasass</groupId>
  <artifactId>recommendation-system</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>pom</packaging>
  <modules>
    <module>eureka-server</module>
    <module>user-service</module>
    <module>video-service</module>
    <module>recommendation-service</module>
    <module>data-processing-service</module>
  </modules>
  ...
</project>

Eureka Service

Add Spring Boot Web and Eureka Server dependencies, then create the main class:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer;

@SpringBootApplication
@EnableEurekaServer
public class EurekaServerApplication {
    public static void main(String[] args) {
        SpringApplication.run(EurekaServerApplication.class, args);
    }
}

Configure application.yml with port 8761 and disable client registration.

server:
  port: 8761
eureka:
  client:
    register-with-eureka: false
    fetch-registry: false
  server:
    enable-self-preservation: false

User Service

Dependencies include Spring Web, Eureka client, and Kafka. The main controller provides a mock history endpoint:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;
import java.util.Arrays;
import java.util.List;

@SpringBootApplication
@EnableEurekaClient
@RestController
public class UserServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(UserServiceApplication.class, args);
    }

    @GetMapping("/users/{userId}/history")
    public List<String> getUserHistory(@PathVariable String userId) {
        return Arrays.asList("1", "3", "5", "7");
    }
}

A Kafka producer service sends user‑behavior messages:

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class KafkaProducerService {
    private static final String TOPIC = "user-behavior-topic";
    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;
    public void sendUserBehavior(String userId, String videoId, String videoTag, int isInterested) {
        String message = String.format("User:%s watched Video:%s [Tag:%s] with interest:%d", userId, videoId, videoTag, isInterested);
        kafkaTemplate.send(TOPIC, message);
    }
}

Controller to record watching actions:

import org.springframework.web.bind.annotation.*;

@RestController
public class UserController {
    @Autowired
    private KafkaProducerService kafkaProducerService;
    @PostMapping("/users/{userId}/watch/{videoId}/{videoTag}/{isInterested}")
    public String watchVideo(@PathVariable String userId,
                            @PathVariable String videoId,
                            @PathVariable String videoTag,
                            @PathVariable int isInterested) {
        kafkaProducerService.sendUserBehavior(userId, videoId, videoTag, isInterested);
        return String.format("User %s watched video %s with tag %s and interest %d", userId, videoId, videoTag, isInterested);
    }
}

Video Service

Provides a static list of videos with tags:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Arrays;
import java.util.List;

@SpringBootApplication
@EnableEurekaClient
@RestController
public class VideoServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(VideoServiceApplication.class, args);
    }
    @GetMapping("/videos")
    public List<Video> getAllVideos() {
        return Arrays.asList(
            new Video("1", "娱乐"),
            new Video("2", "娱乐"),
            new Video("3", "科技"),
            new Video("4", "美食"),
            new Video("5", "科技"),
            new Video("6", "美食"),
            new Video("7", "旅游"),
            new Video("8", "科技")
        );
    }
    static class Video {
        private String id;
        private String tag;
        public Video(String id, String tag) { this.id = id; this.tag = tag; }
        public String getId() { return id; }
        public String getTag() { return tag; }
    }
}

Recommendation Service

Uses Feign clients to call User and Video services, and Redis to fetch the user profile. The core endpoint:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.cloud.openfeign.EnableFeignClients;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.web.bind.annotation.*;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;

@SpringBootApplication
@EnableEurekaClient
@EnableFeignClients
@RestController
public class RecommendationServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(RecommendationServiceApplication.class, args);
    }
    @Autowired
    private StringRedisTemplate redisTemplate;
    @Autowired
    private VideoServiceClient videoServiceClient;
    @GetMapping("/recommendations/{userId}")
    public List<VideoServiceClient.Video> getRecommendations(@PathVariable("userId") String userId) {
        String key = "user:" + userId;
        Set<String> interests = redisTemplate.opsForSet().members(key + ":interests");
        List<String> history = redisTemplate.opsForList().range(key + ":history", 0, -1);
        if (interests.isEmpty() || history.isEmpty()) {
            return videoServiceClient.getAllVideos();
        }
        List<VideoServiceClient.Video> all = videoServiceClient.getAllVideos();
        return all.stream()
            .filter(v -> !history.contains(v.getId()))
            .filter(v -> interests.contains(v.getTag()))
            .collect(Collectors.toList());
    }
}

Feign clients:

@FeignClient(name = "user-service")
public interface UserServiceClient {
    @GetMapping("/users/{userId}/history")
    List<String> getUserHistory(@PathVariable("userId") String userId);
}

@FeignClient(name = "video-service")
public interface VideoServiceClient {
    @GetMapping("/videos")
    List<Video> getAllVideos();
    class Video { private String id; private String tag; /* getters/setters */ }
}

Data‑Processing Service

Consumes the user-behavior-topic Kafka messages, extracts userId, videoId, and tag, and updates Redis sets and lists:

@Service
public class KafkaConsumerService {
    @Autowired
    private DataProcessingService dataProcessingService;
    @KafkaListener(topics = "user-behavior-topic", groupId = "data-processing-group")
    public void consumeUserBehavior(String message) {
        try {
            String userId = extractValue(message, "User:(\\d+)");
            String videoId = extractValue(message, "Video:(\\d+)");
            String videoTag = extractValue(message, "Tag:([^]]+)");
            if (userId != null && videoId != null && videoTag != null) {
                dataProcessingService.processUserBehavior(userId, videoId, videoTag);
            }
        } catch (Exception e) { e.printStackTrace(); }
    }
    private String extractValue(String msg, String pattern) {
        Matcher m = Pattern.compile(pattern).matcher(msg);
        return m.find() ? m.group(1) : null;
    }
}

@Service
public class DataProcessingService {
    @Autowired
    private StringRedisTemplate redisTemplate;
    private static final String PREFIX = "user:";
    private static final String HISTORY = "history";
    private static final String INTERESTS = "interests";
    public void processUserBehavior(String userId, String videoId, String tag) {
        String key = PREFIX + userId;
        redisTemplate.opsForList().leftPush(key + ":" + HISTORY, videoId);
        redisTemplate.opsForSet().add(key + ":" + INTERESTS, tag);
    }
}

Service Degradation

When Redis becomes unavailable, the recommendation endpoint is protected by Resilience4j circuit breaker. The fallback simply returns all videos from the Video service.

@Autowired
private StringRedisTemplate redisTemplate;
@Autowired
private VideoServiceClient videoServiceClient;
private static final String REDIS_BACKEND = "redisService";

@GetMapping("/recommendations/{userId}")
@CircuitBreaker(name = REDIS_BACKEND, fallbackMethod = "getAllVideosFallback")
public List<VideoServiceClient.Video> getRecommendations(@PathVariable("userId") String userId) {
    // same logic as before …
}

public List<VideoServiceClient.Video> getAllVideosFallback(Throwable ex) {
    System.err.println("Redis unavailable, fallback to all videos: " + ex.getMessage());
    return videoServiceClient.getAllVideos();
}

Testing

After starting all services, calling /recommendations/{userId} returns the full list on first request. Recording a watch action via the user service updates Redis; subsequent recommendation calls return only videos matching the user's interests and not yet watched.

Conclusion

The article demonstrates a complete end‑to‑end Spring Cloud microservice recommendation system, covering service decomposition, inter‑service communication with Feign, asynchronous event handling with Kafka, state storage in Redis, and resilience with circuit breakers. While simplified, the pattern scales to real‑world scenarios involving sophisticated recommendation algorithms.