Mastering Go Local Cache: TTL, Sharded LRU, Singleflight & Async Refresh

1️⃣ Cache Analogy

Imagine the database as a warehouse and the cache as shelves: TTL is the expiration date on items, LRU removes the least recently used goods, Sharded LRU partitions shelves to reduce lock contention, Singleflight ensures only one request fetches a missing key, and async refresh silently replenishes items before they expire.

2️⃣ Advanced Design Principles

TTL + LRU : Controls expiration and capacity.

Sharded LRU : Reduces lock competition, improves concurrency.

Singleflight : Prevents cache breakdown by deduplicating concurrent loads.

Async Refresh : Smoothly updates hot data without blocking reads.

Empty Value Placeholder : Stops cache penetration.

Async Persistence : Enables fast recovery after service restart.

Prometheus Metrics : Provides observability and optimization data.

3️⃣ Core Data Structures

type cacheItem struct {
    key        string
    value      interface{}
    expiration int64
    ttl        time.Duration
}

type shard struct {
    mu       sync.Mutex
    items    map[string]*list.Element
    lru      *list.List
    capBytes int64
    curBytes int64
}

type Cache struct {
    shards          []*shard
    group           singleflight.Group
    defaultTTL      time.Duration
    emptyTTL        time.Duration
    refreshFactor   float64
    backgroundPool  chan struct{}
    weigher         Weigher
    persister       Persister
    totalMaxBytes   int64
}

Each shard manages its own LRU list, reducing lock contention; cacheItem.expiration provides precise TTL control; Weigher allows capacity management by object size.

4️⃣ Core Methods

✅ GetOrLoadCtx (Singleflight + Async Refresh)

func (c *Cache) GetOrLoadCtx(ctx context.Context, key string, loader func(context.Context) (interface{}, time.Duration, error)) (interface{}, error) {
    if val, ok := c.Get(key); ok {
        c.asyncRefresh(key, loader, val)
        return val, nil
    }
    v, err, _ := c.group.Do(key, func() (interface{}, error) {
        val, ttl, err := loader(ctx)
        if err != nil {
            return nil, err
        }
        if val == nil {
            ttl = c.emptyTTL
        }
        c.SetWithTTL(key, val, ttl)
        return val, nil
    })
    return v, err
}

Cache hit returns the value immediately and triggers asynchronous refresh.

Cache miss uses singleflight to avoid duplicate loads.

Empty‑value placeholder prevents penetration.

✅ asyncRefresh (Refresh‑After‑Read)

func (c *Cache) asyncRefresh(key string, loader func(context.Context) (interface{}, time.Duration, error), oldVal interface{}) {
    sh := c.getShard(key)
    item, ok := sh.items[key].Value.(*cacheItem)
    if !ok || item.ttl <= 0 {
        return
    }
    remain := time.Until(time.Unix(0, item.expiration))
    if remain < time.Duration(float64(item.ttl)*c.refreshFactor) {
        go func() {
            _, _ = c.group.Do(key, func() (interface{}, error) {
                val, ttl, err := loader(context.Background())
                if err == nil && val != nil {
                    c.SetWithTTL(key, val, ttl)
                }
                return nil, nil
            })
        }()
    }
}

The background goroutine refreshes the entry before its TTL expires, invisible to callers.

5️⃣ Practical Example

package main

import (
    "context"
    "fmt"
    "time"
    "github.com/example/go-local-cache/cache"
)

func main() {
    c := cache.NewCache(cache.CacheOptions{
        DefaultTTL:    5 * time.Second,
        EmptyTTL:      1 * time.Second,
        RefreshFactor: 0.3,
        ShardCount:   8,
        TotalMaxBytes: 1 << 20,
    })
    loader := func(ctx context.Context) (interface{}, time.Duration, error) {
        time.Sleep(10 * time.Millisecond)
        return fmt.Sprintf("val-%d", time.Now().UnixNano()), 5 * time.Second, nil
    }
    for i := 0; i < 5; i++ {
        v, _ := c.GetOrLoadCtx(context.Background(), "user:1001", loader)
        fmt.Println("read:", v)
        time.Sleep(1 * time.Second)
    }
    c.Delete("user:1001")
    c.Purge()
}

6️⃣ Performance Test & Metrics

var wg sync.WaitGroup
numGoroutines := 50
numOps := 200

start := time.Now()
for i := 0; i < numGoroutines; i++ {
    wg.Add(1)
    go func() {
        defer wg.Done()
        for j := 0; j < numOps; j++ {
            key := fmt.Sprintf("key-%d", j%100)
            c.GetOrLoadCtx(context.Background(), key, loader)
        }
    }()
}
wg.Wait()
fmt.Println("Operations completed in:", time.Since(start))
fmt.Println("Shard[0] items:", c.ShardItems(0))
fmt.Println("Shard count:", c.ShardCount())

Key observation points:

Hit rate : Repeated hot keys improve cache utilization.

Async refresh : Triggered when remaining TTL falls below refreshFactor.

LRU eviction : When capacity is reached, old entries are removed.

Prometheus metrics can expose hits, misses, evictions, and refresh counts for clear observability.

7️⃣ Application Scenarios

High‑frequency user profile or permission caching.

Hot product or flash‑sale inventory caching.

Configuration and feature‑flag storage.

Third‑party API response caching.

8️⃣ Optimization Suggestions

Hot‑key pre‑warming: load popular items ahead of traffic.

Batch back‑source: reduce round‑trip calls.

Cache degradation: fall back to stale data or defaults when fresh data is unavailable.

Staggered refresh: apply different refresh strategies per key type.

Persisted recovery: background persistence enables fast cache warm‑up after restart.

Conclusion

By combining sharded LRU, TTL, singleflight, and asynchronous refresh, this Go local cache delivers high‑concurrency safety, controllable capacity, hot‑key protection, full observability via Prometheus, and operational friendliness through delete, purge, and persistence features, making it ready for production services.

Source code repositories:

GitHub: https://github.com/louis-xie-programmer/go-local-cache.git

Gitee: https://gitee.com/louis_xie/go-local-cache.git