How JD Retail's Firefly AIGC Platform Scales AI Content Production with Unified AI Infra

Why Content Production Needs AI Infra

As AIGC shifts from one-off generation to large-scale content production, four core engineering problems emerge:

Capability organization : A single content production task involves knowledge-base retrieval, product parsing, image understanding, image generation, copywriting, and post-processing. When model and algorithm capabilities exist as isolated services or ad-hoc scripts, they cannot be uniformly combined, reused, or managed.

Task execution : Online requests demand low latency, batch production demands high throughput, and massive material processing demands elastic resources. CPU preprocessing, GPU inference, and models with high initialization costs have different runtime characteristics; a single execution model cannot cover all scenarios.

Data support for model quality : Model output depends not only on the model but also on material quality, feature completeness, and the ability to retrieve suitable multimodal assets. Raw multimodal data must be processed, governed, and settled before becoming production-ready inputs.

Long-chain observability : A task comprises multiple operators. When failures occur, the platform must know execution progress, which operator failed, and the associated logs and root causes.

To address these, the platform organizes model and algorithm capabilities uniformly, selects execution engines per task profile, processes raw multimodal assets into searchable data, and establishes workflow-wide observability — together forming an AI Infra for content production.

Overall Architecture: Unified Orchestration, Differentiated Execution

The architecture revolves around two synergistic pipelines:

Content production pipeline : Business side (Firefly) combines AIGC capabilities via Agent, Workflow, and production templates. The platform layer parses these into executable Pipeline/DAGs and schedules them to execution engines — Ray Serve, Ray Data, or Spark/Flink — based on task characteristics.

Multimodal data processing pipeline : Raw images and videos enter Lance multimodal tables. Elastic compute performs inference tagging, content understanding, feature processing, and cleansing. Results, tags, and features are written back to the lakehouse; qualified assets then enter image/video knowledge bases to provide retrieval for content production.

Trace horizontally connects the platform layer and execution engines, using DAG, task IDs, operator IDs, and logs to reconstruct actual execution flows.

Platform Layer: From Business Workflow to Executable DAG

Operator Management: Turning Algorithm Capabilities into Platform Execution Units

An operator is the smallest schedulable capability unit — image processing logic, a model inference capability, or a standardized external service call. The platform constrains inputs/outputs, resource declarations, versions, business parameters, and dependency environments uniformly, so upper-layer Workflows need not know algorithm internals.

Decoupling model weights, business parameters, runtime configs, and core code lets the same capability be upgraded independently or routed to different runtimes per task type. Operators become platform execution units with contract, version, and resource semantics, not one-off scripts.

Pipeline/DAG: Compiling Capabilities into Execution Plans

The platform parses selected Workflow capabilities into a Pipeline/DAG describing operator dependencies, execution order, and data flow. Dependent nodes run sequentially; independent nodes run in parallel. Upstream/downstream data passes via input-output mapping — downstream operators fetch only needed fields, outputs can be filtered, renamed, and mapped, avoiding direct coupling between algorithm implementations and unbounded intermediate data propagation.

Task Management: From Execution Plan to Concrete Runtime

After DAG generation, the platform creates a task linked to the Workflow, operator versions, business config, and resource requirements. Task management provides submission, config, and status views; a unified orchestration DSL describes tasks and selects execution backends. The platform unifies task, operator, and data-flow semantics while runtimes remain differentiated: online model serving, offline batch inference, and multimodal data processing route to Ray Serve, Ray Data, and elastic compute respectively.

Execution System: Real-time, Batch, and Elastic

Different AIGC tasks have distinct latency, throughput, model lifecycle, and compute requirements. Firefly does not force one runtime for all; under unified task semantics it routes tasks to real-time, batch, or elastic paths per task profile.

Why Ray

Ray provides three key abstractions matching AI workloads: stateless Tasks, stateful Actors for model reuse, and Ray Serve for online model replicas. These suit high model initialization cost, long runtimes, and heterogeneous resources. For Python-centric models and operators, Ray offers unified resource semantics — operators declare CPU, GPU, memory, concurrency, and node labels; scheduling places image preprocessing, text processing, and GPU inference on appropriate nodes. Online and offline tasks share one distributed execution system but use Ray Serve and Ray Data runtimes respectively. Ray's value lies not in single-inference performance but in unifying model serving, service replicas, resource scheduling, and distributed data processing under one consistent execution interface for upper-layer Pipelines.

Ray Serve: Managing Online Model Service Lifecycle

Real-time tasks are scheduled by a custom engine and invoke Ray Serve-deployed operators via HTTP. Serve provides:

Model residency and service replicas : Models and operators pre-load in resident instances, avoiding per-request re-initialization. Each service independently configures replica count, CPU/GPU resources, and concurrency; scaling one model does not require scaling the entire Workflow.

Rolling deployment and auto-scaling : New versions roll out gradually to reduce online impact; replica count adjusts automatically to load peaks and valleys.

Decoupled orchestration and model runtime : Custom scheduler handles task orchestration and request routing; Ray Serve handles model loading, replicas, request dispatch, and runtime resources. They connect via stable HTTP interfaces, leaving room for multi-model collaboration and complex Workflows.

Ray Data: Offline Batch Processing for Model Inference

Batch content production uses Ray Data for offline inference and distributed data processing. Input data is organized as Datasets, distributed across model instances; read, inference, and write form a continuous data pipeline. This suits large-scale batch inference with Python models better than per-item calls.

Model instance reuse : Operator Actors run with long lifecycles; models load once at Actor init and process multiple batches. Actor Pools maintain reusable model instances, cutting repeated initialization cost for image and multimodal models.

Batch inference and resource coordination : map_batches sends a batch to model instances, reducing per-call scheduling overhead and aligning with GPU inference patterns. Each operator independently configures batch size, concurrency, CPU/GPU, memory, and node labels, balancing throughput, VRAM usage, and isolation.

Pipelined parallel execution : Data read, model processing, and write advance in parallel as data flows; independent branches (e.g., image understanding, attribute extraction) run in parallel and merge results.

Ray Data transforms offline inference from "loop-calling models" to "distributed execution over datasets and model instances."

Elastic Compute

Elastic compute handles massive multimodal data lakehouse tasks — image/video cleansing, inference tagging, content understanding, feature processing. These exhibit periodic compute peaks; isolating them on dedicated elastic resources prevents large-scale data processing from impacting online content generation and daily batch production.

The core design: unified task semantics, differentiated execution. The platform layer describes tasks uniformly; each engine optimizes its runtime for online latency, batch throughput, or data scale.

Multimodal Data Lakehouse: From Asset Governance to Data Flywheel

Model quality depends on input accuracy, asset relevance, and context completeness. For content production, raw images/videos must undergo understanding, tagging, feature extraction, and quality cleansing before becoming searchable, reusable knowledge-base assets.

Why Lance

Multimodal data differs from traditional analytics data. One asset record may link image/video, structured metadata, model tags, understanding results, and feature vectors; as algorithms iterate, new tags, features, and scoring columns continuously accrue.

Lance, designed for AI data access, unifies images, videos, structured metadata, inference tags, understanding results, features, and vectors in one tabular format. Three factors drove its adoption:

Columnar organization and on-demand reads : Metadata, tags, understanding results, features, and vectors are column-organized. Tasks read only needed columns, suiting wide multimodal tables and ever-growing feature columns.

Unified association of assets and inference results : Raw assets, model results, tags, and vectors link within one data organization. New features or processing results extend around the same asset data, reducing cross-system copying and joining.

Vector search and fast point lookup : Assets can be filtered by structured tags/attributes or retrieved via vector search for semantic similarity; hits allow fast asset metadata reads, providing a unified access path for visual inspection and knowledge-base construction.

Thus Lance tables serve as both the unified multimodal asset carrier and the shared data entry for batch processing, visual inspection, and knowledge-base building.

Asset Cleansing: From Lake Ingestion to Knowledge Base

The current lakehouse centers on a main asset-processing and knowledge-base pipeline, with a visualization platform as a sidecar for data observation and manual operations:

Business, algorithm, and ops personnel view datasets, inference results, tags, and features across the lakehouse.

Filter by tags, attributes, processing results; vector-search for related data.

Use for verifying cleansing results, observing distributions, spotting anomalies.

Qualified assets enter image/video knowledge bases; content production tasks retrieve models, on-body images, and video assets from there.

Front-loading high-cost image/video understanding and feature extraction lets production directly use governed multimodal assets, reducing ad-hoc processing pressure in online Workflows.

Data Flywheel: From Data Accumulation to Model and Operator Iteration

Built capabilities cover asset ingestion, elastic processing, result settlement, visual observation, and knowledge-base construction. The next goal is to gradually settle inference results and problem samples from content production into a complete data flywheel:

Connect data accumulation, issue analysis, and model iteration.

Make online production not just consume model capabilities but also accumulate data for the next optimization round.

As this matures, the multimodal lakehouse evolves from an asset governance space into a data asset space serving retrieval, evaluation, training, and Agent context simultaneously.

Trace: Making Workflow Execution Visible

A content production Pipeline may contain multiple sequentially dependent or parallel operators. Scattered logs make it hard to quickly determine execution progress and failure location. Trace connects DAG structure, tasks, operators, and logs into a workflow-level execution view.

Task, Operator, and Log Association Mechanism

Trace uses unified identifiers to stitch one task's execution: task_id identifies a complete task; DAG provides workflow structure. operator_id maps to a specific operator in the flow. trace_id links the operator to actual runtime logs.

The platform can drill from task overview to DAG, then from a node to relevant logs without manual cross-system ID matching.

What Can Be Observed

Trace shows whether tasks and operators executed, success/failure status, and on failure pinpoints the specific operator, error context, and associated logs. It serves business users tracking progress and algorithm/platform engineers understanding long-chain runtime behavior.

On top of existing workflow observability, future work includes automatic task retry and intermediate result recovery.

Conclusion: Let Models, Data, and Tasks Collaborate in One System

Firefly AI Infra forms a clear technical chain for multimodal content production: the platform layer converts business Workflows into executable DAGs; Ray Serve, Ray Data, and elastic compute route tasks to suitable runtimes; Lance multimodal lakehouse turns raw assets into searchable knowledge-base data; Trace connects tasks, operators, and logs into a complete execution view.

The key is not stacking components but making unified orchestration, differentiated execution, upfront data governance, and workflow-level observability collaborate. Business continuously composes new AIGC capabilities; the platform standardizes them into tasks; engines allocate resources per task profile; multimodal data continuously settles and evolves; task execution becomes observable and debuggable. Thus model capabilities move from pointwise calls to continuous, stable content production and delivery.