Build a Scalable TikTok‑Style Recommendation System with Spring Cloud Microservices

1. Introduction

Douyin's recommendation system is a key to its success, backed by a complex micro‑service architecture that handles high concurrency and massive user data. Building an efficient, scalable micro‑service architecture is essential for such scenarios.

This article uses Spring Cloud to construct a simplified Douyin recommendation system, exploring micro‑service design and practice.

2. Business Requirements

User Behavior Data

The core of a recommendation system is personalized recommendation, which requires comprehensive user behavior data (views, likes, shares, comments, etc.). The system must:

Record user behavior data : capture interactions such as video views and likes.

Manage user profiles : generate interest profiles from historical behavior for recommendation calculations.

Video Resource Management

Douyin must manage a large number of short videos, each with tags (type, topic, style) that feed the recommendation algorithm. The system needs:

Store basic video information : ID, title, tags, upload time, etc.

Provide video categorization : classify videos based on tags for later recommendation.

Personalized Recommendation

Generate a personalized list by matching user interest profiles with video tags:

Obtain user profile and video tags : match videos that may interest the user.

Generate recommendation list : compute and return a personalized video list.

Real‑time User Behavior Processing

User actions occur in real time, so the system must process these events promptly and update user profiles:

Real‑time processing : receive events such as likes or views and update the profile instantly.

Asynchronous handling : use message queues to decouple processing from the recommendation service.

Summarizing the core functions:

User behavior management: record views, likes, etc.

Video resource management: store basic info and tags.

Personalized recommendation: combine user profile and video tags.

Real‑time data processing: update user profile on the fly.

3. Architecture Design

Four core functions are abstracted into four services. The simplified architecture diagram (API gateway omitted) shows each service registered with Eureka and communication via Feign. Redis caches user profiles; Kafka transports user behavior events.

4. Implementation Details

Project Setup

Use Maven archetype to create a multi‑module project. The parent pom.xml manages modules: eureka-server, user-service, video-service, recommendation-service, and data-processing-service. Add Spring Boot 2.6.4 and Spring Cloud 2021.0.1 dependencies.

<project xmlns="http://maven.apache.org/POM/4.0.0" ...>
  <modelVersion>4.0.0</modelVersion>
  <modules>
    <module>eureka-server</module>
    <module>user-service</module>
    <module>video-service</module>
    <module>recommendation-service</module>
    <module>data-processing-service</module>
  </modules>
  ...
</project>

Create Eureka Service

Add spring-cloud-starter-netflix-eureka-server dependency, create EurekaServerApplication with @EnableEurekaServer, and configure application.yml (port 8761).

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer;

@SpringBootApplication
@EnableEurekaServer
public class EurekaServerApplication {
    public static void main(String[] args) {
        SpringApplication.run(EurekaServerApplication.class, args);
    }
}

Run the application and access http://localhost:8761 to see the Eureka console.

Create User Service

Add web starter, Eureka client, and Kafka dependencies. Register the service to Eureka (port 8081) and expose a REST endpoint to return mock user history.

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;
import java.util.Arrays;
import java.util.List;

@SpringBootApplication
@EnableEurekaClient
@RestController
public class UserServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(UserServiceApplication.class, args);
    }

    @GetMapping("/users/{userId}/history")
    public List<String> getUserHistory(@PathVariable String userId) {
        return Arrays.asList("1", "3", "5", "7");
    }
}

Implement a Kafka producer to send user behavior events:

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class KafkaProducerService {
    private static final String TOPIC = "user-behavior-topic";
    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    /**
     * Send user behavior to Kafka
     */
    public void sendUserBehavior(String userId, String videoId, String videoTag, int isInterested) {
        String message = String.format("User:%s watched Video:%s [Tag:%s] with interest:%d", userId, videoId, videoTag, isInterested);
        kafkaTemplate.send(TOPIC, message);
    }
}

Expose an endpoint to record watching behavior and forward it to Kafka:

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class UserController {
    @Autowired
    private KafkaProducerService kafkaProducerService;

    @PostMapping("/users/{userId}/watch/{videoId}/{videoTag}/{isInterested}")
    public String watchVideo(@PathVariable String userId,
                             @PathVariable String videoId,
                             @PathVariable String videoTag,
                             @PathVariable int isInterested) {
        kafkaProducerService.sendUserBehavior(userId, videoId, videoTag, isInterested);
        return String.format("User %s watched video %s with tag %s and interest %d", userId, videoId, videoTag, isInterested);
    }
}

Create Video Service

Add web starter and Eureka client, expose /videos returning a mock list of videos with IDs and tags.

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Arrays;
import java.util.List;

@SpringBootApplication
@EnableEurekaClient
@RestController
public class VideoServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(VideoServiceApplication.class, args);
    }

    @GetMapping("/videos")
    public List<Video> getAllVideos() {
        return Arrays.asList(
            new Video("1", "娱乐"),
            new Video("2", "娱乐"),
            new Video("3", "科技"),
            new Video("4", "美食"),
            new Video("5", "科技"),
            new Video("6", "美食"),
            new Video("7", "旅游"),
            new Video("8", "科技")
        );
    }

    static class Video {
        private String id;
        private String tag;
        public Video(String id, String tag) { this.id = id; this.tag = tag; }
        public String getId() { return id; }
        public String getTag() { return tag; }
    }
}

Create Recommendation Service

Add web starter, Eureka client, Feign, Redis, and Jackson dependencies. Define Feign clients for user and video services, a Redis repository for user profiles, and the main controller that reads interests and history from Redis, fetches all videos via Feign, and filters according to the profile.

import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import java.util.List;

@FeignClient(name = "user-service")
public interface UserServiceClient {
    @GetMapping("/users/{userId}/history")
    List<String> getUserHistory(@PathVariable("userId") String userId);
}

@FeignClient(name = "video-service")
public interface VideoServiceClient {
    @GetMapping("/videos")
    List<Video> getAllVideos();
    class Video { private String id; private String tag; public String getId() { return id; } public String getTag() { return tag; } }
}

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.cloud.openfeign.EnableFeignClients;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;

@SpringBootApplication
@EnableEurekaClient
@EnableFeignClients
@RestController
public class RecommendationServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(RecommendationServiceApplication.class, args);
    }

    @Autowired
    private StringRedisTemplate redisTemplate;
    @Autowired
    private VideoServiceClient videoServiceClient;

    @GetMapping("/recommendations/{userId}")
    public List<VideoServiceClient.Video> getRecommendations(@PathVariable("userId") String userId) {
        String key = "user:" + userId;
        Set<String> interests = redisTemplate.opsForSet().members(key + ":interests");
        List<String> history = redisTemplate.opsForList().range(key + ":history", 0, -1);
        if (interests == null || interests.isEmpty() || history == null || history.isEmpty()) {
            return videoServiceClient.getAllVideos();
        }
        List<VideoServiceClient.Video> all = videoServiceClient.getAllVideos();
        return all.stream()
            .filter(v -> !history.contains(v.getId()))
            .filter(v -> interests.contains(v.getTag()))
            .collect(Collectors.toList());
    }
}

Create Data‑Processing Service

Add Spring Kafka, Redis, Jackson, and Eureka client dependencies. Configure Kafka consumer (topic user-behavior-topic) and Redis connection. Implement a listener that parses the message, extracts userId, videoId, and tag, then updates Redis: push videoId to a list representing history and add tag to a set representing interests.

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;
import java.util.regex.*;

@Service
public class KafkaConsumerService {
    @KafkaListener(topics = "user-behavior-topic", groupId = "data-processing-group")
    public void consume(String message) {
        String userId = extract(message, "User:(\\d+)");
        String videoId = extract(message, "Video:(\\d+)");
        String tag = extract(message, "Tag:([^]]+)");
        if (userId != null && videoId != null && tag != null) {
            // delegate to processing service
        }
    }
    private String extract(String msg, String regex) {
        Matcher m = Pattern.compile(regex).matcher(msg);
        return m.find() ? m.group(1) : null;
    }
}

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;

@Service
public class DataProcessingService {
    @Autowired
    private StringRedisTemplate redisTemplate;
    private static final String PREFIX = "user:";
    private static final String HISTORY = "history";
    private static final String INTERESTS = "interests";

    public void process(String userId, String videoId, String tag) {
        String key = PREFIX + userId;
        redisTemplate.opsForList().leftPush(key + ":" + HISTORY, videoId);
        redisTemplate.opsForSet().add(key + ":" + INTERESTS, tag);
    }
}

5. Business Testing

Initially, calling /recommendations/{userId} returns all videos because no profile exists. After invoking the watch endpoint (e.g., user watches video 3 with interest flag), the Kafka consumer updates Redis. Subsequent recommendation calls return only videos matching the user's interests and not previously watched (e.g., IDs 5 and 8 for a tech‑interested user).

6. Service Degradation (Circuit Breaker)

To avoid total failure when a single service (e.g., Redis) is down, integrate Resilience4j circuit breaker. Add the dependency spring-cloud-starter-circuitbreaker-resilience4j and configure a breaker named redisService with a sliding window of 10 calls, 50% failure threshold, 10‑second open state, etc.

resilience4j:
  circuitbreaker:
    instances:
      redisService:
        slidingWindowSize: 10
        failureRateThreshold: 50
        waitDurationInOpenState: 10000
        permittedNumberOfCallsInHalfOpenState: 3
        minimumNumberOfCalls: 5
        automaticTransitionFromOpenToHalfOpenEnabled: true
        slowCallDurationThreshold: 2000
        slowCallRateThreshold: 50

Modify the recommendation controller to use @CircuitBreaker and provide a fallback that directly calls the video service when Redis is unavailable.

import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;

@GetMapping("/recommendations/{userId}")
@CircuitBreaker(name = "redisService", fallbackMethod = "fallbackAllVideos")
public List<VideoServiceClient.Video> getRecommendations(@PathVariable String userId) {
    // existing logic that reads from Redis
}

public List<VideoServiceClient.Video> fallbackAllVideos(Throwable ex) {
    System.err.println("Redis unavailable: " + ex.getMessage());
    return videoServiceClient.getAllVideos();
}

Stopping Redis and invoking the recommendation endpoint triggers the fallback, logs the error, and returns the full video list, demonstrating graceful degradation.

7. Conclusion

The article shows how to build a simple recommendation system with Spring Cloud microservices, covering service decomposition, inter‑service communication via Feign, asynchronous event handling with Kafka, profile caching in Redis, and resilience using Resilience4j. While the demo is simplified, the same principles apply to real‑world, large‑scale recommendation platforms.