Cache Redesign and Performance Optimization for Community Collection Service

The community collection feature experiences read‑heavy, write‑light traffic. Initial cache design showed no issues under low load, but monitoring revealed high response times (RT) and excessive Redis/MySQL QPS.

Problem analysis : The "is collection" API had RT up to 8 ms, far above the expected 1‑2 ms Redis latency. Redis and MySQL QPS were amplified 15‑fold, with MySQL accounting for ~37 % of upstream traffic, indicating low cache hit rate and frequent fallback queries.

Pseudo‑code before optimization :

//判断用户是否对指定的动态收藏
func IsLightContent(userId uint64,contentIds []uint64){
    index := userId%20
    cacheKey := key + "_" + fmt.Sprintf("%d", index)
    pipe := redis.GetClient().Pipeline()
    for _, item := range contentIds {
        InitCache(userId, contentId)
        pipe.SisMember(cacheKey, userId)
    }
    pipe.Exec()
    //......
}

//缓存初始化判断，不存在则初始化数据缓存
func InitCache(userId uint64,contentId uint64){
    index := userId%20
    cacheKey := key + "_" + fmt.Sprintf("%d", index)
    ttl,_ := redis.GetClient().TTL(cacheKey)
    if ttl <= 0{ //key不存在或者未设置过期时间
        // query from db
        // save to redis
    }else{
        redis.GetClient().Expire(cacheKey,time.Hour()*48)
    }
}

The code iterates over content IDs, creates 20 sharded keys per user, and checks membership with SISMEMBER, causing many Redis calls and large key space.

Solution direction : Reduce Redis calls per request and avoid sharding. Two main paths were considered:

Replace per‑content iteration with a single batch query.

Store a user's collected content in a single key rather than many sharded sets.

Because a user’s collection size is moderate, the cache can store the entire set for active users. For heavy users, a limit of the most recent 5,000 content IDs (sorted descending) is cached; IDs beyond this are fetched from MySQL.

Implementation changes:

Switch cache structure from SET to HASH with TTL extended to 7 days.

Replace SISMEMBER checks with a single HMGET call, halving Redis accesses.

Results (data from a 7‑day window):

API response RT improved ~3× and stabilized.

Redis QPS for TTL, SISMEMBER, and HMGET dropped to near zero.

Redis memory usage and key count decreased ~3×.

MySQL SELECT QPS reduced ~24×; connection concurrency fell ~3×, reducing wait times.

The redesign demonstrates how proper cache design dramatically improves performance and resource utilization for backend services.