Feeder Pipelines: Bridging Offline Data Factory to Online Recommendation Engine
This article details Feeder pipelines that transport diverse data types—user features, content tags, embeddings, popularity rankings, and operational rules—from offline computation to online serving, covering batch atomic switches, NRT incremental updates, cursor-based recovery, scheduling platform design, and latency observability practices.
1. Where Online System Data Comes From
Online recommendation systems consume data produced offline: user profiles, content features, statistical features, trained models, etc. The Feeder is the bridge that moves this data from the offline data factory (batch/stream pipelines) into online stores (Feature Server, Elasticsearch, Milvus, Redis). It is not a single pipeline but a set of pipelines, each tailored to the data's volume, update frequency, and consistency requirements.
2. Data Categories Handled by Feeder
User Profile / User Features : Written to Feature Server. Daily full batch update with dual-version atomic switch; 5-minute NRT incremental updates via Redis atomic writes (SET/HSET/INCR) with deduplication.
Content Features / Other Features : Same pattern as user features — daily batch + 5-minute NRT.
Content Tags : Written to Elasticsearch inverted index for tag-based recall. Daily incremental scan using a cursor (lastUpdateTime) stored in Redis; per-document update via ES document update API. Atomicity at document level; full index atomic switch not required.
Content Embeddings : Written to Milvus vector index for vector recall. Daily full rebuild: create new collection, load vectors, build index, verify, then switch alias. Old collection retained for rollback. If embedding model changes, both query and content vectors must stay in compatible space; alias switch alone does not coordinate this.
Popularity Data : Written to Redis (dozens of keys, each a ranked item list). Updated every 5 minutes or hourly by full key replacement. Tolerates brief cross-key mixed state; atomic batch switch possible but not used initially.
Operational Data (CMS) : Blacklists, pinned/boosted items. Source is CMS system. Feeder uses "broadcast + full sync": consumes Kafka real-time change events (with timestamps) for low-latency propagation, plus daily full pull for reconciliation. Timestamp-based versioning prevents stale overwrites; delete events or snapshot diff handle removals.
3. Feeder Pipeline Details
3.1 Feature Feeder Pipeline
Sources: offline Feature Pipeline (batch long-term features, NRT short-term features). Batch: dual-version atomic switch in Feature Server (two versions, one serving, one receiving; switch after validation). NRT: calls Feature Server's RealtimeUpdate interface; uses Redis atomic ops with deduplication (INCR not idempotent, so must pair with dedup keys or use overwrite with versioned full results). Batch and NRT for same feature must coordinate coverage boundaries (baseline cutoff, incremental window, late-arrival compensation) and switch as a logical version together.
3.2 Tag Inverted Index Feeder Pipeline
Daily incremental scan from Content Pool (initially HDFS dump, later API over HBase). Cursor = scan start timestamp saved in Redis per datacenter. On success, cursor advances; on failure, cursor not updated, enabling retry from last point. Cursor loss falls back to full scan. Requires idempotent writes, stable pagination, and handling of simultaneous timestamp boundaries and delayed visibility.
3.3 Vector Index Feeder Pipeline
Daily full rebuild in Milvus: new collection → write vectors → build index → load → verify → alias switch. Chosen because rebuild cost controllable and full index simplifies validation/rollback. Milvus supports upsert but incremental logic adds complexity. Model version changes require compatible query/content vector spaces; alias switch does not enforce this.
3.4 Popularity Feeder Pipeline
Offline Popularity Pipeline computes rankings periodically; Feeder syncs to local Redis per datacenter. Dozens of keys, each a full list. Per-key replacement; brief mixed state tolerated. Atomic batch switch (dual-version) possible if business demands.
3.5 CMS Feeder Pipeline
Combines Kafka broadcast (real-time) + daily full pull (reconciliation). Both paths carry timestamps; writes use timestamp comparison at target atomically to avoid stale overwrites. Deletes propagated via delete events or snapshot diff. Broadcast ensures SLA for time-sensitive ops (e.g., content takedown); full pull corrects drift, not a substitute for retry/alert on broadcast failures.
4. Feeder Organization & Scheduling Platform
4.1 Initial Approach
Two pipeline types: periodic (Feature, Tag, Vector, Popularity) via crontab; continuous (CMS) as long-running process consuming Kafka. Simple but lacked observability, unified monitoring/alerting, and governance.
4.2 Scheduling Platform
Central platform manages all pipelines: periodic scheduled, continuous launched and monitored. Pipelines expose runtime state nodes (e.g., batch: fetch → load candidate → validate → publish; incremental: fetch → validate → write → confirm → advance cursor; CMS: key events). Platform handles scheduling, retries, staggering, alerts, history, state detection (overlapping runs, stuck tasks). Business logic (e.g., Tag cursor) remains inside pipeline; only checkpoints exposed.
4.3 Declarative Pipeline Configuration
Each pipeline defined by source, target, mode, cursor, schedule, validation rules. Example YAML:
pipeline: tag-inverted
type: batch
source: content-pool-api
source_tag_version: v1.1
mode: incremental
cursor: redis:local:tag_last_scan
target: es-doc-update
other_params: xxx
schedule: 0 2 * * *
validate: [row_count, null_rate]Standardized adapters allow new pipelines via config only.
4.4 Layered Implementation & Open-Source Alternatives
Four layers: Config (declarative YAML), Orchestration (parsing, scheduling, retries, alerts; batch vs resident tasks), Sync Framework (generic fetch/batch/write/retry/cursor advance), Source/Target Adapters (Content Pool API, ES doc update, Redis, Milvus, CMS API, Kafka consumer). Sync framework and adapters are core investment. Today, DolphinScheduler/Airflow could handle batch orchestration; SeaTunnel offers connectors for batch/stream sync. However, publish semantics (dual-version switch, NRT-baseline coordination, cursor recovery) still need custom integration regardless of orchestration tool.
5. Core Engineering Problems
5.1 From Transfer Complete to Online Available: Version Release Chain
Data written ≠ immediately usable. Full chain for vector index: upstream content ready → fetch/transform → write new collection → build index → validate → load/warmup → readiness probe → alias switch → online query verification. Feeder must wait for target service states (e.g., ES refresh, Milvus load) per publish contract; independent probes confirm final visibility.
5.2 Incremental Cursor & Resume
Tag Feeder uses scan-start timestamp as cursor. Overlap rescans cover in-window changes but not sufficient alone: need inclusive boundary re-scan + idempotent writes, stable pagination, upstream snapshot or change log for mid-scan mutations, and late-arrival watermark or re-scan window. Cursor persisted only after task success; same principle for NRT batches and CMS Kafka offsets. Write success = target ack (not necessarily search visibility). Idempotency and stale-write prevention essential under retries.
5.3 Feature Batch & NRT: Coordinated Coverage Switch
Batch baseline covers events up to 00:00 but loads at 02:00; NRT continues processing 00:00–02:00. Switch must include: baseline coverage range, incremental range, late-arrival compensation rule. Both baseline and matching increment prepared, then switched as one logical version. Rollback restores old baseline, its increment range, feature config, and model contract. Detailed boundary logic referenced in Article 7 §3.4.
5.4 Atomicity & Validation: Designed by Business Constraints
Summary of consistency boundaries:
ES inverted index: per-doc atomic; progressive visibility ok; search visibility confirmed separately.
Milvus vector: full version built, alias switch; query/index compatibility required.
Redis popularity: per-key full replace; brief cross-key mix tolerated.
Batch Feature: service-layer version switch; per-request fixed version; cache version-isolated.
Realtime Feature: per-entity/window update; prevent stale overwrite & replay duplicate; coordinate related fields.
Operational data: per-record async + full reconciliation; meet propagation SLA; related rules published as group.
Pre-publish validation on dual-version sets: row count, null rate, schema/dimension, version compatibility, distribution drift, sample diff. Failures block switch, alert, manual investigate. Invalid records go to replayable dead-letter queue, not silently dropped. Cursor advance decision based on criticality; time-sensitive changes (takedowns) alert and track until resolved.
5.5 Latency Observability: Success ≠ Freshness
Four latency metrics:
Sync duration: task start → target write done.
Feeder publish latency: upstream artifact ready → online confirmed available (includes scheduling, sync, validate, load, switch).
End-to-end latency: source event/modification → online available (includes upstream production).
Data freshness: now − online data coverage time. A successful task may publish stale data.
Record timestamps at each stage (upstream ready, task start, write done, load ready, online confirmed). Total latency = difference of corresponding endpoints; stage durations for bottleneck detection (stages may overlap or have hidden waits). Alert on publish latency, freshness, and stalled progress per data cadence. Online verification probes: special ES document periodically updated and searched to detect visibility lag; vector index query to confirm serving collection version and its coverage time; Feature Server returns active batch version and NRT coverage watermark. Rewriting old data updates timestamps without improving freshness — probes must check coverage time, not write time. Real incident: Tag Feeder hung for hours due to Content Pool timeouts; retries without timeout budget or progress monitoring delayed detection. Lesson: retries need timeout budget, backoff, progress monitoring; exhaust budget → explicit failure, preserve resume point, alert.
6. Conclusion
Feeder connects offline data factory (Articles 4–9) and online recommendation engine (Articles 10–15). It does more than move data — it advances offline artifacts to online usability. Different data types demand different publish semantics: incremental doc updates (ES), full-version alias switch (Milvus), per-key replace (Redis popularity), coordinated batch+NRT (Feature), broadcast+reconcile (CMS). Pipeline designs are driven by data characteristics and business tolerance. With this article, the offline–online data channel is complete. Next article (18) reviews architectural evolution across scale, latency, and team changes. Subsequent engineering practice series will cover request tracing (Reader) and config/operational change propagation.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
