Designing a Trillion‑Scale Feed: Hybrid Push‑Pull Architecture and Kubernetes in Production

Why Feed Systems Reach the "Push‑or‑Pull" Limit

Feed is a core infrastructure for social, content, e‑commerce, and news products. Users expect a sorted list of posts within tens to hundreds of milliseconds. When DAU grows to tens of millions and creators reach hundreds of millions of posts per day, the system evolves from a simple "lookup‑follow‑list + fetch‑content" to a highly complex distributed service where write amplification, read amplification, storage redundancy, network traffic, and operational volatility all explode.

Feed as a Timeline Assembly System

The feed service does not store content; it assembles a timeline from four logical layers:

Content Layer : stores dynamic media metadata, visibility, audit status.

Relationship Layer : stores follows, blocks, social graph.

Delivery Layer : decides if and when a post enters a user’s readable set.

Read Layer : merges, deduplicates, filters, sorts, paginates and returns the result.

Two logical views are maintained:

Outbox – author’s publishing perspective ("what I posted").

Inbox / Timeline – reader’s consumption perspective ("what I can see").

Push, Pull, and Hybrid Principles

Push (write‑time materialization)

When an author publishes, the post is pre‑written into every follower’s Inbox.

Pros: ultra‑short read path, simple sorting/pagination (usually a Redis ZSET), stable experience for ordinary users.

Cons: write amplification O(fans_count), massive fan‑out for celebrity authors, high storage redundancy, many ineffective deliveries to inactive fans.

Complexity: write O(fans_count), read O(page_size).

Pull (read‑time materialization)

Authors only write to their Outbox; readers aggregate the latest posts from the authors they follow at request time.

Pros: lightweight writes O(1), low storage redundancy, handles hot authors well.

Cons: long read path, heavy relationship queries, costly K‑way merge, unstable latency for long follow lists.

Complexity: write O(1), read O(following_count × recent_window × merge_cost).

Hybrid (selective write‑ and read‑side materialization)

Pure push or pull rarely works at scale. A mature hybrid system applies multi‑dimensional policies:

Author fan‑size tiering (L1/L2/L3).

Reader activity tiering (active, semi‑active, lazy).

Content freshness tiering.

Dynamic load‑aware thresholds.

Business‑scenario routing (home page, pagination, notifications, detail view).

The core of hybrid is strategy‑based routing rather than a simple if‑else on fan count.

Capacity Modeling – Why Designs That Look Good on Paper Fail in Production

A typical large‑scale social platform assumes:

DAU = 80 million

Peak online = 12 million

Daily active authors = 6 million

Daily posts = 1.8 billion

Peak publish QPS = 25 k

Peak feed request QPS = 450 k

Average follows per user = 280

Big V (celebrity) share = 0.1 %

Super‑big V fan count = 1 M–5 M

Pure push for a 20 M‑fan author would require 20 M Inbox writes, each costing ~0.05 ms, consuming hundreds of megabytes of memory for a single post. A burst of ten such authors would saturate Redis, network, and Kafka partitions.

Pure pull for a normal user following 500 accounts and fetching the latest 20 items yields a candidate set of 10 k, requiring expensive aggregation, deduplication, and sorting even if the candidates are cached.

Optimization Objective

Minimize:
  total_cost = write_amp_cost + read_amp_cost + storage_cost + cache_miss_cost + failure_recovery_cost
Subject to:
  publish_p99 < 200 ms
  first_page_p99 < 150 ms
  timeline_consistency acceptable
  system_availability >= 99.95 %

The system must answer five concrete questions:

How to cap write fan‑out?

How to absorb read hot‑spots with cache or pre‑compute?

How to tier hot/cold indexes?

How to degrade gracefully when components jitter?

How to scale on Kubernetes without causing secondary avalanches?

Production‑Grade Hybrid Architecture

The service is split into clear responsibilities: Publish Command Service: validates, writes feed_content, creates a transactional Outbox record. Event Bus / Kafka: reliable fan‑out channel. Fanout Worker Cluster: decides Push / Lazy‑Push / Skip, slices fan‑out, writes to Inbox. Inbox Cache Cluster (Redis/Dragonfly): hot timeline index. Inbox Storage Tier: warm/cold timeline persistence. Social Graph DB: follows, blocks, user tags. Feed Aggregator: merges Push and Pull results for the client.

Key Design Principles (Eight Things That Really Matter)

Deliver only to "worth‑delivering" users – filter by recent activity (1 day strong push, 7 days weak push, >30 days lazy or pull).

Model big V with three tiers (L1 full push, L2 push to active fans, L3 pure pull with optional hot‑pool pre‑compute).

Inbox stores only indexes (feed_id, author_id, publish_time, rank_score, visibility_bits); full content stays in the content layer.

Writes must be idempotent – use feed_id as ZSET member and rank_score as score; duplicate writes produce no extra members.

Read path tolerates eventual consistency – most posts become visible within 1–3 seconds; strict strong consistency is unnecessary.

Prioritize first‑page latency – hot Inbox + limited Pull for the initial page, deeper pages may fall back to cold storage.

Strategy engine must include hysteresis and versioning to avoid oscillation when fan count hovers around thresholds.

Assume component jitter (Kafka hot partitions, Redis hot keys, pod restarts, DB slow queries, config lag) and provide full rate‑limit, circuit‑break, degradation, replay, and compensation mechanisms.

Data Model Details

SQL schema examples (MySQL/TiDB) illustrate separation of content, follows, author profile, and outbox events. Indexes on (author_id, created_at DESC) and (status, created_at DESC) support recent‑author queries.

Redis Timeline Structures

Inbox hot index: feed:inbox:{userId} → ZSET (member = feedId, score = rankScore).

Author outbox hot cache: feed:outbox:{authorId} → ZSET (member = feedId, score = publishTime).

Active‑user tags: SET or BITMAP feed:active:user:{bucket}.

Idempotent slice marker: feed:fanout:done:{feedId}:{sliceId} (TTL 7‑30 days).

ZSET is chosen for its ordered, paginable, and idempotent properties; alternative warm/cold layers can use List + compression or columnar stores, but any replacement must keep fast truncation, pagination, idempotent update, and rank‑based ordering.

Publishing Pipeline – From Transactional Consistency to Scalable Fan‑Out

Receive publish request.

Validate content and permissions.

Insert into feed_content.

Insert a transactional Outbox event ( feed_outbox_event).

Commit transaction.

Outbox Relay scans ready records (using SKIP LOCKED or pessimistic lock) and publishes to Kafka.

Fanout Workers consume, evaluate strategy, slice fan‑out, and write to Inbox via Redis pipeline.

Metrics are reported; failures trigger exponential back‑off retries.

Sample Spring‑Boot Publish Service

package com.example.feed.application;

import com.example.feed.domain.*;
import com.example.feed.infrastructure.id.IdGenerator;
import jakarta.transaction.Transactional;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import java.time.Instant;

@Service
@RequiredArgsConstructor
public class FeedPublishService {
    private final FeedContentRepository feedContentRepository;
    private final FeedOutboxEventRepository outboxEventRepository;
    private final AuthorProfileRepository authorProfileRepository;
    private final IdGenerator idGenerator;

    @Transactional
    public PublishResult publish(PublishCommand command) {
        validate(command);
        long feedId = idGenerator.nextId();
        long eventId = idGenerator.nextId();
        Instant now = Instant.now();
        AuthorStrategy strategy = authorProfileRepository.loadStrategy(command.authorId());
        FeedContent content = FeedContent.builder()
                .feedId(feedId)
                .authorId(command.authorId())
                .contentType(command.contentType())
                .contentRef(command.contentRef())
                .visibilityType(VisibilityType.PUBLIC.getCode())
                .status(1)
                .createdAt(now)
                .updatedAt(now)
                .build();
        FeedOutboxEvent event = FeedOutboxEvent.builder()
                .eventId(eventId)
                .feedId(feedId)
                .authorId(command.authorId())
                .eventType(1)
                .status(0)
                .retryCount(0)
                .nextRetryAt(now)
                .payloadJson(PublishPayload.of(feedId, command.authorId(), strategy).toJson())
                .createdAt(now)
                .updatedAt(now)
                .build();
        feedContentRepository.insert(content);
        outboxEventRepository.insert(event);
        return new PublishResult(feedId, strategy.mode().name(), now);
    }

    private void validate(PublishCommand command) {
        if (command.authorId() == null || command.authorId() <= 0) {
            throw new IllegalArgumentException("invalid authorId");
        }
        if (command.contentRef() == null || command.contentRef().isBlank()) {
            throw new IllegalArgumentException("contentRef is empty");
        }
    }
}

Outbox Relay (Database → Kafka)

package com.example.feed.infrastructure.relay;

import com.example.feed.infrastructure.messaging.KafkaPublisher;
import com.example.feed.infrastructure.persistence.FeedOutboxEventRepository;
import com.example.feed.infrastructure.persistence.model.OutboxRecord;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.time.Instant;
import java.util.List;

@Slf4j
@Component
@RequiredArgsConstructor
public class FeedOutboxRelay {
    private static final int BATCH_SIZE = 500;
    private final FeedOutboxEventRepository repository;
    private final KafkaPublisher kafkaPublisher;

    @Scheduled(fixedDelay = 200)
    public void relay() {
        List<OutboxRecord> records = repository.lockReadyRecords(BATCH_SIZE, Instant.now());
        for (OutboxRecord record : records) {
            try {
                kafkaPublisher.publish("feed-publish", String.valueOf(record.getFeedId()), record.getPayloadJson());
                repository.markSuccess(record.getEventId(), Instant.now());
            } catch (Exception ex) {
                log.error("relay failed, eventId={}", record.getEventId(), ex);
                repository.markRetry(record.getEventId(), record.getRetryCount() + 1,
                        Instant.now().plusSeconds(backoffSeconds(record.getRetryCount())));
            }
        }
    }

    private long backoffSeconds(int retryCount) {
        return Math.min(300, 1L << Math.min(retryCount, 8));
    }
}

Fanout Worker – Core Responsibilities

package com.example.feed.worker;

import com.example.feed.worker.model.FanoutEvent;
import com.example.feed.worker.model.FanoutSlice;
import com.example.feed.worker.service.*;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.stereotype.Component;

@Slf4j
@Component
@RequiredArgsConstructor
public class FanoutWorker {
    private final FanoutStrategyService strategyService;
    private final FollowerQueryService followerQueryService;
    private final InboxWriteService inboxWriteService;

    @KafkaListener(topics = "feed-publish", groupId = "feed-fanout-worker", concurrency = "12")
    public void onMessage(FanoutEvent event, Acknowledgment ack) {
        try {
            FanoutDecision decision = strategyService.evaluate(event);
            if (decision.skipPush()) {
                ack.acknowledge();
                return;
            }
            List<FanoutSlice> slices = followerQueryService.loadSlices(event.authorId(), decision.activeOnly(), decision.sliceSize());
            for (FanoutSlice slice : slices) {
                inboxWriteService.writeSlice(event.feedId(), event.authorId(), event.rankScore(), slice);
            }
            ack.acknowledge();
        } catch (Exception ex) {
            log.error("fanout failed, feedId={}", event.feedId(), ex);
            throw ex;
        }
    }
}

Inbox Write Service – Slice‑Level Idempotent Batch

package com.example.feed.worker.service;

import com.example.feed.worker.model.FanoutSlice;
import lombok.RequiredArgsConstructor;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.nio.charset.StandardCharsets;
import java.time.Duration;

@Service
@RequiredArgsConstructor
public class InboxWriteService {
    private static final int MAX_TIMELINE_SIZE = 800;
    private final StringRedisTemplate redisTemplate;

    public void writeSlice(long feedId, long authorId, double rankScore, FanoutSlice slice) {
        String doneKey = "feed:fanout:done:" + feedId + ":" + slice.sliceId();
        Boolean first = redisTemplate.opsForValue().setIfAbsent(doneKey, "1", Duration.ofDays(7));
        if (Boolean.FALSE.equals(first)) {
            return; // slice already processed
        }
        redisTemplate.executePipelined((RedisCallback<Object>) connection -> {
            for (Long userId : slice.userIds()) {
                byte[] inboxKey = key("feed:inbox:" + userId);
                byte[] member = bytes(String.valueOf(feedId));
                connection.zAdd(inboxKey, rankScore, member);
                connection.zRemRange(inboxKey, 0, -(MAX_TIMELINE_SIZE + 1));
            }
            return null;
        });
    }

    private byte[] key(String key) { return key.getBytes(StandardCharsets.UTF_8); }
    private byte[] bytes(String value) { return value.getBytes(StandardCharsets.UTF_8); }
}

Read Path – Merging Push and Pull Within 100 ms

The read service follows four steps:

Load top entries from the hot Inbox (Redis ZSET).

Identify Pull authors (big V, recent active authors) for the user.

Fetch latest candidates from each author’s Outbox.

K‑way merge, dedup, fill missing slots, and materialize full objects.

Sample Java implementation uses a priority queue (max‑heap) of OutboxCursor objects, merges up to three times the page size, deduplicates while preserving order, then materializes visible items via a batch DB query.

public TimelinePage firstPage(long userId, int pageSize) {
    List<Long> inboxFeedIds = inboxQueryRepository.loadTopFeedIds(userId, pageSize * 2);
    List<Long> pullAuthors = pullAuthorRepository.loadPullAuthors(userId, 200);
    PriorityQueue<OutboxCursor> heap = new PriorityQueue<>(Comparator.comparing(OutboxCursor::publishTime).reversed());
    for (Long authorId : pullAuthors) {
        authorOutboxRepository.loadLatest(authorId, 3)
                .stream().findFirst().ifPresent(heap::offer);
    }
    List<Long> merged = new ArrayList<>(pageSize * 3);
    merged.addAll(inboxFeedIds);
    while (!heap.isEmpty() && merged.size() < pageSize * 3) {
        OutboxCursor cur = heap.poll();
        merged.add(cur.feedId());
        authorOutboxRepository.loadNext(cur.authorId(), cur.publishTime(), 1)
                .stream().findFirst().ifPresent(heap::offer);
    }
    List<Long> deduped = dedupKeepOrder(merged);
    List<FeedItemView> items = feedMaterializeRepository.loadVisibleByIds(userId, deduped, pageSize);
    Instant nextCursor = items.isEmpty() ? Instant.now() : items.get(items.size() - 1).publishTime();
    return new TimelinePage(items, nextCursor);
}

Ranking & Scoring – Beyond Simple Chronology

Production feeds combine time, quality, relationship strength, interaction history, content score, compliance risk, and decay for duplicate suppression. A two‑stage ranking is typical:

Recall ranking : lightweight score = timeWeight + relationshipBoost + interactionPrior – penalty.

Fine‑ranking : adds author affinity, click‑through rate, dwell time, content quality, risk score, and duplicate decay.

Common Pitfalls (10 Real‑World Gotchas)

Strategy oscillation when fan count hovers around thresholds – solved with hysteresis, versioning, and delayed config rollout.

Wasting resources on completely inactive fans – filter by activity tags (1 day, 7 days, 30 days).

First‑page under‑fill after filtering – over‑sample candidate set and tune first‑page vs pagination separately.

Redis hot keys for super‑active users – hash‑shard keys, cache hot author outbox locally, add near‑cache.

Kafka partition skew – primary topic ordered by authorId, secondary fanout topic sharded by slice hash.

Timeline cursor instability – use cursor based on timestamp + feedId instead of offset.

Stale deletions – broadcast delete events, filter at materialization.

Follow‑change dirty reads – verify follow relationship at read time and clean old Inbox entries asynchronously.

Scaling pods without scaling Redis/DB – perform end‑to‑end bottleneck analysis before HPA.

Lack of performance baseline – maintain standard load‑test models per author tier, activity level, and peak event.

Engineering Evolution – Service Decomposition

feed‑command‑service

: publish, delete, pin, permission changes. feed‑query‑service: home page, pagination, detail lookup. fanout‑worker: asynchronous fan‑out. graph‑service: follow/follower queries. ranking‑service: scoring and strategy evaluation. timeline‑repair‑job: compensation, back‑fill, audit.

Storage Tiering

Relational metadata (MySQL/TiDB) – sharded.

Hot timeline index – Redis Cluster.

Cold timeline – Cassandra/ScyllaDB/HBase.

Content objects – object storage with metadata table.

Kubernetes Production Practices

Deployments must consider pod churn impact on Kafka rebalancing, correct HPA metrics, graceful termination, and avoidance of capacity drops during rolling updates.

API Service Deployment Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: feed-query-service
  labels:
    app: feed-query-service
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: feed-query-service
  template:
    metadata:
      labels:
        app: feed-query-service
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: app
        image: registry.example.com/feed-query-service:1.0.0
        ports:
        - containerPort: 8080
        env:
        - name: JAVA_TOOL_OPTIONS
          value: "-XX:+UseContainerSupport -XX:MaxRAMPercentage=70"
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 20
          periodSeconds: 10
        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "sleep 20"]
        resources:
          requests:
            cpu: "1000m"
            memory: "2Gi"
          limits:
            cpu: "4000m"
            memory: "6Gi"

Fanout Worker Deployment Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: feed-fanout-worker
spec:
  replicas: 12
  selector:
    matchLabels:
      app: feed-fanout-worker
  template:
    metadata:
      labels:
        app: feed-fanout-worker
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: worker
        image: registry.example.com/feed-fanout-worker:1.0.0
        env:
        - name: SPRING_KAFKA_LISTENER_ACK_MODE
          value: MANUAL_IMMEDIATE
        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "sleep 30"]
        resources:
          requests:
            cpu: "2000m"
            memory: "4Gi"
          limits:
            cpu: "6000m"
            memory: "8Gi"

HPA Based on Business Metrics

Instead of pure CPU, the fanout worker HPA watches Kafka lag and CPU utilization, with a long scale‑down window to avoid thrashing during traffic spikes.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: feed-fanout-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: feed-fanout-worker
  minReplicas: 12
  maxReplicas: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 20
        periodSeconds: 60
  metrics:
  - type: Pods
    pods:
      metric:
        name: kafka_consumer_lag
      target:
        type: AverageValue
        averageValue: "1500"
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Monitoring – Full‑Stack Observability

Publish side: QPS, success rate, Outbox backlog, relay failure rate, retry latency.

Fanout side: Kafka lag, slice count per event, slice execution latency, Redis pipeline RTT, skip rate, compensation rate.

Read side: Home‑page P50/P90/P99, Pull author count distribution, candidate set size distribution, materialize hit rate, first‑page fill rate.

Storage side: Redis memory water‑mark, hot key Top‑N, relational DB slow queries, cold storage scan volume.

Business side: post‑visibility within 1 s / 5 s, first‑page blank rate, follow‑page click‑through.

Case Study – Protecting the System During a Celebrity Post

A star with 48 M fans posts during a New Year event. Pure push would flood Redis with millions of writes, overflow Kafka, and saturate worker CPU.

Production hybrid marks the author as L3 (pure Pull) and only pushes to a small core of highly active fans (those who interacted within the last day). The rest receive the post via Pull. Hot author outbox is cached locally and in Redis, reducing latency for active fans while avoiding a write storm.

Evolution Roadmap – From Zero to Trillion‑Scale

Stage 1 – Monolith or simple microservice: feed_content + Redis Inbox, basic Push, few big V Pull.

Stage 2 – Standard Hybrid: Outbox + Kafka + Fanout Worker, active‑user tiering, pre‑computed Pull author sets.

Stage 3 – Strong Engineering Hybrid: cold/hot timeline separation, full service split, HPA on business metrics, compensation jobs.

Stage 4 – Global multi‑region: replicated Outbox, active‑active data centers, near‑read routing.

Practical Recommendations for Architects

Decouple publish transaction from fan‑out using transactional Outbox.

Layer users – push for ordinary users, Pull for celebrities, lazy‑push for semi‑active fans.

Stop full Push for low‑activity users immediately to save resources.

Make the home‑page read path a combination of hot Inbox + limited Pull.

Implement compensation, audit, and replay before adding complex personalization.

Finally, add dynamic thresholds, personalized ranking, edge caching as refinements.

In summary, push, pull, and hybrid are not mutually exclusive; they are complementary tools applied to different user segments and load conditions. A well‑engineered hybrid feed delivers low‑latency experience for ordinary users, protects the system from celebrity‑induced storms, and provides a cost‑optimal, observable, and evolvable architecture.