Skip to content
From Generation to Production: Firefly AIGC Platform's AI Infra Practice

From Generation to Production: Firefly AIGC Platform's AI Infra Practice

The article details how the Firefly AIGC platform evolved from a chat interface to a production-grade content generation system, covering its unified orchestration architecture, differentiated execution using Ray Serve and Ray Data, multimodal data lakehouse with Lance, and workflow observability via Trace.

JD Cloud Developers
JD Cloud Developers
JD Cloud Developers

Introduction: From a Single Request to a Production System

Many AIGC products start as a chat window where users input requests and models return text or images. However, when generation results enter actual content production, the challenge shifts from whether a single model call succeeds to whether multiple capabilities can be stably organized to continuously complete content generation and delivery.

Firefly is a content-production-oriented AIGC platform that includes content Agents, Workflows, production templates, model capabilities, and browser operations. Users can organize content production via dialogue, workflows, or templates, and combine the Firefly browser extension for web operations, content publishing, and multi-platform distribution.

For a user, a request may be "generate a product图文 content"; for the platform, it requires knowledge-base retrieval, image understanding, model inference, copywriting generation, result processing, and content delivery. Different stages depend on different models, data, and compute resources, and must be orchestrated into an executable and observable task chain.

Therefore, Firefly's AI Infra focuses not only on whether models can generate results, but on how models, data, tasks, and compute collaborate to form a stable content production system.

Why Content Production Needs AI Infra

After AIGC moves from single-generation to scaled content production, the system must solve four core engineering problems:

How capabilities are organized : A single content production often includes knowledge-base retrieval, product parsing, image understanding, image generation, copywriting generation, and result processing. If model and algorithm capabilities remain as independent services or ad-hoc scripts, they are hard to uniformly combine, reuse, and manage.

How tasks are executed : Online requests care about latency, batch production cares about throughput, massive material processing cares about resource elasticity. CPU preprocessing, GPU inference, and high-initialization-cost models have different runtime characteristics; a single execution mode cannot cover all scenarios.

How data supports model effectiveness : Model output depends not only on the model itself but also on material quality, feature completeness, and the ability to retrieve suitable multimodal materials. Raw multimodal data must be processed, governed, and precipitated before becoming directly usable inputs for production.

How long chains are observed : A task consists of multiple operators. When exceptions occur, the platform needs to know where execution reached, which operator failed, and the corresponding logs and failure reasons.

To address these, the platform needs a unified way to organize model and algorithm capabilities, select execution engines based on task characteristics, process raw multimodal materials into retrievable data, and establish workflow-wide observability. These capabilities together constitute the AI Infra for content production.

Overall Architecture: Unified Orchestration, Differentiated Execution

Overall architecture diagram showing content production pipeline and multimodal data processing pipeline
Overall architecture diagram showing content production pipeline and multimodal data processing pipeline

The overall architecture revolves around two collaborative pipelines. The content production pipeline starts from the Firefly business layer: the business side selects and combines AIGC capabilities via Agent, Workflow, and production templates; the platform layer parses these capabilities into executable Pipeline/DAGs, then schedules them to execution engines such as Ray Serve, Ray Data, or Spark/Flink based on task characteristics.

The other pipeline is multimodal data processing. Raw materials like images and videos enter Lance multimodal tables, where elastic compute performs inference labeling, content understanding, feature processing, and cleaning. Inference results, labels, and features are written back and precipitated in the lakehouse. After screening, qualified materials enter image/video knowledge bases, providing retrieval capabilities for content production.

Trace horizontally connects the platform layer and different execution engines, restoring the actual execution process of tasks through DAG, task identifiers, operator identifiers, and logs.

Platform Layer: From Business Workflow to Executable DAG

Platform layer architecture showing operator management, Pipeline/DAG compilation, and task management
Platform layer architecture showing operator management, Pipeline/DAG compilation, and task management

1. Operator Management: Turning Algorithm Capabilities into Platform Execution Units

An operator is the smallest capability unit that the platform can recognize and schedule. It can be a piece of image processing logic, a model inference capability, or a standardized call to an external service. Through operator management, the platform uniformly constrains inputs/outputs, resource declarations, versions, business parameters, and dependency environments, so upper-layer Workflows do not need to know how algorithms are implemented internally.

After decoupling model weights, business parameters, runtime configurations, and core code, the same capability can be upgraded independently or enter different runtime modes based on task type. Operators thus become platform execution units with contract, version, and resource semantics, rather than one-off scripts for a specific requirement.

2. Pipeline/DAG: Compiling Capabilities into Execution Plans

The platform parses the capabilities selected in a Workflow into a Pipeline/DAG. The DAG describes operator dependencies, execution order, and data flow: nodes dependent on upstream results execute sequentially, while nodes without dependencies can run in parallel.

Data is passed between upstream and downstream via input-output mapping. Downstream operators only fetch the fields they need. Operator outputs can be filtered, renamed, and mapped to subsequent nodes, avoiding direct coupling between different algorithm implementations and preventing unbounded intermediate data propagation in long chains.

3. Task Management: From Execution Plan to Concrete Runtime

After DAG generation, the platform creates a corresponding task, associating the Workflow, operator versions, business configurations, and resource requirements used in this run. Task management provides entry points for task submission, runtime configuration, and status viewing. A unified orchestration DSL describes tasks and selects execution backends.

The platform unifies task, operator, and data-flow semantics, while runtimes can remain differentiated. Online model services, offline batch inference, and multimodal data processing respectively enter Ray Serve, Ray Data, and elastic compute.

Execution System: Real-time, Batch, and Elastic

Different AIGC tasks have distinct requirements for latency, throughput, model lifecycle, and compute resources. Firefly does not use a single runtime for all tasks; instead, under unified task semantics, it selects real-time, batch, or elastic execution paths based on task characteristics.

Execution system diagram showing Ray Serve, Ray Data, and elastic compute paths
Execution system diagram showing Ray Serve, Ray Data, and elastic compute paths

1. Why Choose Ray

Ray provides three important abstractions simultaneously: Task, Actor, and serviced model instances. Stateless computations can run as Tasks; stateful or model-reuse computations can be carried by Actors; online models can run as service replicas via Ray Serve. These abstractions match AI task characteristics such as high model initialization cost, long runtime, and complex resource types.

For Python-centric models and algorithm operators, Ray offers unified resource semantics. Operators can declare CPU, GPU, memory, concurrency, and machine label requirements; scheduling then places image preprocessing, text processing, and GPU inference onto appropriate resource nodes. Online and offline tasks share a distributed execution system but adopt Ray Serve and Ray Data runtime models respectively.

Ray's value is not just single-point inference performance, but organizing model serving, service replicas, resource scheduling, and distributed data processing into a single system, providing a consistent execution interface for upper-layer Pipelines.

2. Ray Serve: Managing Online Model Service Lifecycle

Real-time tasks are scheduled by a custom scheduling engine and invoke operators deployed in Ray Serve via HTTP interfaces. Serve provides three key capabilities:

Model residency and service replicas : Models and operators are pre-loaded in resident instances, avoiding repeated initialization per request. Different services can independently configure replica counts, CPU/GPU resources, and concurrency; resource changes for a single model do not require scaling the entire Workflow.

Rolling deployment and auto-scaling : When models or operators release new versions, Serve supports gradual instance switching, reducing version-switch impact on online requests. It also adjusts replica counts based on service load to adapt to request peaks and valleys.

Decoupling platform orchestration from model runtime : The custom scheduling engine handles task orchestration and request scheduling; Ray Serve handles model loading, service replicas, request dispatch, and runtime resources. They connect via stable HTTP interfaces, leaving room for future multi-model collaboration and complex Workflows.

3. Ray Data: Offline Batch Processing for Model Inference

Batch content production uses Ray Data for offline inference and distributed data processing. Pending data is organized as Datasets and distributed to different model instances for execution; reading, inference, and result writing are chained into a continuous data processing flow. Compared to per-item model calls, this approach better suits large-scale batch inference with Python models.

Model instance reuse : Model operators run as long-lived Actors; models load once during Actor initialization and continuously process multiple data batches. An Actor Pool maintains a group of reusable model instances, reducing repeated initialization costs for image and multimodal models.

Batch inference and resource collaboration : map_batches sends a batch of data to model instances for centralized processing, reducing per-call scheduling overhead and better matching GPU inference execution patterns. Different operators can separately configure batch size, concurrency, CPU/GPU, memory, and machine labels, balancing throughput, VRAM usage, and resource isolation.

Pipelined parallel execution : Data reading, model processing, and result writing can advance in parallel as data flows; independent branches like image understanding and attribute extraction in long chains can also run in parallel, merging results after completion.

Ray Data transforms offline inference from "loop-calling models" into "distributed execution oriented to datasets and model instances."

4. Elastic Compute

Elastic compute handles massive material processing in the multimodal data lakehouse, including image/video cleaning, inference labeling, content understanding, and feature processing. These tasks typically have phased compute peaks; placing them in independent elastic resources reduces the impact of large-scale data processing on online content generation and daily batch production.

The core of this design is unified task semantics with differentiated execution: the platform layer uniformly describes tasks, while execution engines optimize their own runtime for online latency, batch throughput, and data scale respectively.

Multimodal Data Lakehouse: From Material Governance to Data Flywheel

Model effectiveness depends not only on the model but also on input data accuracy, material matching, and context completeness. For content production, raw images and videos must undergo understanding, labeling, feature extraction, and quality cleaning before becoming retrievable, reusable materials in knowledge bases.

1. Why Choose Lance

Multimodal data differs from traditional analytical data. A material record may simultaneously associate images/videos, structured metadata, model labels, content understanding results, and feature vectors. As algorithms iterate, the same data continuously gains new labels, features, and scoring columns.

Lance is designed for AI data access scenarios, able to organize images, videos, structured metadata, inference labels, content understanding results, features, and vectors in a single tabular format. For multimodal material processing, three aspects were key:

Columnar organization and on-demand reading : Material metadata, labels, content understanding results, features, and vectors are organized by column. Processing tasks can read only needed fields, suiting wide multimodal tables and continuously added feature columns.

Unified association of materials and inference results : Raw materials, model inference results, labels, and vectors establish associations within a single data organization. When algorithms add new features or processing results, they can extend around the same material data, reducing cross-system data copying and joining.

Vector search and fast point lookup : Materials can be filtered by structured conditions like labels and attributes, or retrieved via vector search for semantically similar images/videos. After retrieval hits, corresponding material information can be quickly read, providing a unified access path for visual screening and knowledge base construction.

Thus, Lance tables serve as both the unified carrier for multimodal materials and the shared data entry for subsequent batch processing, visual screening, and knowledge base construction.

2. Material Cleaning: From Material Ingestion to Knowledge Base Construction

The current multimodal data lakehouse centers on a main pipeline of material processing and knowledge base construction, with a lakehouse visualization platform as a side entry connecting Lance multimodal tables, supporting data observation and manual operations:

Material cleaning pipeline and visualization platform
Material cleaning pipeline and visualization platform

Business, algorithm, and operations personnel can use the visualization platform to view material datasets, inference results, labels, and features across the lakehouse. They can filter materials by labels, attributes, and processing results, or use vector search to find related data, for checking cleaning results, observing data distributions, and discovering anomalous samples. Governed and screened qualified materials enter image/video knowledge bases, and content production tasks retrieve models, try-on images, and video materials from the knowledge bases.

By front-loading high-cost processing like image/video understanding and feature extraction, the production phase can directly use governed multimodal materials, reducing ad-hoc processing pressure on online Workflows.

3. Data Flywheel: From Data Precipitation to Model and Operator Iteration

Current capabilities cover material ingestion, elastic processing, result precipitation, visual observation, and knowledge base construction. On this basis, the goal is to gradually precipitate inference results and problematic samples from content production, forming a complete data flywheel:

Data flywheel diagram connecting data precipitation, problem analysis, and model iteration
Data flywheel diagram connecting data precipitation, problem analysis, and model iteration

This target chain connects data precipitation, problem analysis, and model iteration, making online production not only consume model capabilities but also accumulate data needed for the next round of optimization. As this capability matures, the multimodal lakehouse can evolve from a material governance space into a data asset space serving retrieval, evaluation, training, and Agent context simultaneously.

Trace: Making Workflow Execution Visible

A content production Pipeline may contain multiple operators with sequential dependencies or parallel execution. Relying solely on scattered logs makes it hard to quickly determine where a task executed to and which node failed. Trace connects DAG structure, tasks, operators, and logs into a workflow-level execution view.

1. Association Mechanism for Tasks, Operators, and Logs

Trace uses unified identifiers to link a task's execution process:

Trace association mechanism showing task_id, operator_id, trace_id
Trace association mechanism showing task_id, operator_id, trace_id
task_id

identifies a complete task; the DAG provides workflow structure; operator_id corresponds to specific operators in the flow; trace_id associates operators with actual runtime logs. The platform can navigate from task overview to DAG, then drill down from specific nodes to relevant logs, without manually correlating identifiers across multiple systems.

2. What Can Be Observed

Trace shows whether tasks and operators executed, whether execution succeeded or failed, and locates the specific operator, error context, and associated logs on failure. It serves both business personnel viewing task progress and algorithm/platform personnel understanding actual long-chain task runtime.

On top of existing workflow observability, future work can add task auto-retry, intermediate result recovery, and other runtime management capabilities.

Conclusion: Letting Models, Data, and Tasks Collaborate in One System

Firefly AI Infra has formed a clear technical chain for multimodal content production: the platform layer converts business Workflows into executable DAGs; Ray Serve, Ray Data, and elastic compute route different tasks to suitable runtimes; the Lance multimodal data lakehouse processes raw materials into retrievable knowledge base data; Trace connects tasks, operators, and logs into a complete execution view.

The key lies not in stacking components, but in forming collaboration among unified orchestration, differentiated execution, data front-loaded governance, and workflow-level observability. Business can continuously combine new AIGC capabilities; the platform converts them into standard tasks; execution engines allocate resources per task characteristics; multimodal data continuously precipitates and evolves; task execution becomes observable and locatable. Thus, model capabilities move from single-point calls to continuous, stable content production and delivery.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

observabilityAIGCRayAI infrastructureContent ProductionWorkflow OrchestrationLanceRay ServeRay DataMultimodal Data Lakehouse
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.