Designing a Scalable Feed System with Redis: Init, Push/Pull, and Code

Background

Recently a requirement emerged to display the activity feed of followed users, which involves designing a feed‑flow system. This article presents a generally applicable solution for enterprises.

Related Concepts

Feed: each status or message in a feed stream, e.g., a Weibo post. Feed Stream: a continuously updated information flow presented to users, such as a personal timeline or a follow page.

Feed Types

Timeline : ordered by publish time; each feed is important and should be shown, similar to WeChat Moments.

Rank : ordered by non‑time factors such as user preferences, typical for news or product recommendation.

Design

Designing a feed system involves two key steps: initialization and push. The storage core uses MySQL for persistence, with Redis for fast access.

Feed Initialization

When a user’s feed does not exist, create a personal feed by iterating the follow list and storing feed IDs in a Redis sorted set (zSet). Key points:

Load initial data from the database.

Use the user ID as the zSet key.

For Timeline, use the feed creation timestamp as the score; for Rank, use a business weight.

Push

After initialization, the feed must be updated in four scenarios: new feed from a followed user, deletion of a followed user’s feed, a user adding a new follow, and a user unfollowing.

Push/Pull Hybrid Model

When a user publishes a new feed:

Read the user’s fan list and determine if the user is a “big V”.

Write the feed to the user’s personal timeline; for big V this ends the process.

If the user is ordinary, write the feed to each fan’s feed (e.g., 100 fans → 100 writes).

When refreshing a personal feed:

Read the list of followed big V users.

Read the user’s own feed.

If big V users exist, concurrently read each big V’s timeline.

Merge results, sort by time, and return.

Pure Push Model

All users follow the same three‑step process: write the feed to the user’s timeline and to each fan’s feed, eliminating the extra read step and reducing network overhead.

Model Comparison

The hybrid model can overload big V timelines during refresh. Solutions include using push for active fans and pull for inactive fans, or adopting a full push model at the cost of higher storage and longer fan‑out time.

Implementation

The author implements a pure push model suitable for a typical enterprise. The implementation consists of three parts: feed initialization, handling publish/delete events, and handling follow/unfollow actions.

Initialize Feed

/**
 * Get the focus feed of a user
 */
public List<FeedDto> listFocusFeed(Long userId, Integer page, Integer size) {
    String focusFeedKey = "focusFeedKey" + userId;
    if (!zSetRedisTemplate.exists(focusFeedKey)) {
        initFocusIdeaSet(userId);
    }
    Double zscore = zSetRedisTemplate.zscore(focusFeedKey, "0");
    if (zscore != null && zscore > 0) {
        return null;
    }
    int offset = (page - 1) * size;
    long score = System.currentTimeMillis();
    List<String> list = zSetRedisTemplate.zrevrangeByScore(focusFeedKey, score, 0, offset, size);
    List<FeedDto> result = new ArrayList<>();
    if (QlchatUtil.isNotEmpty(list)) {
        for (String s : list) {
            FeedDto feedDto = this.loadCache(Long.valueOf(s));
            if (feedDto != null) {
                result.add(feedDto);
            }
        }
    }
    return result;
}

/**
 * Initialize the focus feed zSet
 */
private void initFocusFeedSet(Long userId) {
    String focusFeedKey = "focusFeedKey" + userId;
    zSetRedisTemplate.del(focusIdeaKey);
    List<Feed> list = this.feedMapper.listFocusFeed(userId);
    if (QlchatUtil.isEmpty(list)) {
        zSetRedisTemplate.zadd(focusFeedKey, 1, "0");
        zSetRedisTemplate.expire(focusFeedKey, RedisKeyConstants.ONE_MINUTE * 5);
        return;
    }
    for (Feed feed : list) {
        zSetRedisTemplate.zadd(focusFeedKey, feed.getCreateTime().getTime(), feed.getId().toString());
    }
    zSetRedisTemplate.expire(focusFeedKey, 60 * 60 * 60);
}

Handle Publish/Delete

/**
 * Process fan feeds when a feed is added or removed
 */
public void handleFeed(Long userId, Long feedId, String type) {
    Integer currentPage = 1;
    Integer size = 1000;
    List<FansDto> fansDtos;
    while (true) {
        Page page = new Page();
        page.setSize(size);
        page.setPage(currentPage);
        fansDtos = this.fansService.listFans(userId, page);
        for (FansDto fansDto : fansDtos) {
            String focusFeedKey = "focusFeedKey" + userId;
            if (!this.zSetRedisTemplate.exists(focusFeedKey)) {
                continue;
            }
            if ("feed_add".equals(type)) {
                this.removeOldestZset(focusFeedKey);
                zSetRedisTemplate.zadd(focusFeedKey, System.currentTimeMillis(), feedId);
            } else if ("feed_sub".equals(type)) {
                zSetRedisTemplate.zrem(focusFeedKey, feedId);
            }
        }
        if (fansDtos.size() < size) {
            break;
        }
        currentPage++;
    }
}

/**
 * Remove the oldest entry from a zSet when it exceeds 1000 items
 */
private void removeOldestZset(String focusFeedKey) {
    if (this.zSetRedisTemplate.zcard(focusFeedKey) >= 1000) {
        List<String> zrevrange = this.zSetRedisTemplate.zrevrange(focusFeedKey, -1, -1, String.class);
        if (QlchatUtil.isNotEmpty(zrevrange)) {
            this.zSetRedisTemplate.zrem(focusFeedKey, zrevrange.get(0));
        }
    }
}

Handle Follow/Unfollow

/**
 * Update the follower's zSet when following or unfollowing
 */
public void handleFocus(Long followId, Long followerId, String type) {
    String focusFeedKey = "focusFeedKey" + followerId;
    if (!this.zSetRedisTemplate.exists(focusFeedKey)) {
        return;
    }
    List<FeedDto> feedDtos = this.listFeedByFollowId(source, followId);
    for (FeedDto feedDto : feedDtos) {
        if ("focus".equals(type)) {
            this.removeOldestZset(focusFeedKey);
            this.zSetRedisTemplate.zadd(focusFeedKey, feedDto.getCreateTime().getTime(), feedDto.getId());
        } else if ("unfocus".equals(type)) {
            this.zSetRedisTemplate.zrem(focusFeedKey, feedDto.getId());
        }
    }
}

The code snippets illustrate the core logic; a complete production system would require additional classes and error handling.

Conclusion

A simple, enterprise‑ready feed system has been presented. Feedback and suggestions are welcomed.