Building Industrial-Scale Unstructured Data Pipelines for LLMs: From Text to Multimodal

1. Text Data Processing

Six practical steps convert raw HTML pages into training corpora: (1) acquire original HTML under compliance, (2) parse and clean content, (3) add feature tags (URL, timestamps, domain, language), (4) standardize by deduplication, detoxification, and de‑biasing, (5) publish data and assign a quality rating (A‑D), and (6) run ablation experiments on a small model to verify that the new data improves benchmark scores.

2. Multimodal Data Processing

2.1 VQA

Two generation paths are used: directly query a search engine for an image‑question pair, or synthesize a question and answer from an existing image using a stronger model.

2.2 Caption

Models generate concise, detailed descriptions of images; multiple questions can be posed per image to enrich the training signal.

2.3 Interleave

Text and images are alternated in a single context so the model learns to follow a narrative that mixes visual and textual information, mirroring natural web page layouts.

2.4 Audio

Audio data follows the same pattern as images, with Audio Question Answering (AQA) replacing VQA and similar caption/interleave techniques; quality criteria include language, dialect, genre, and emotional tone.

3. Data‑Engineering Infrastructure

Unstructured assets (audio, video, images) are stored in OSS, while their metadata lives in ODPS; the system is migrating to a unified Paimon lake. Each asset receives a global UID that links all extracted features.

Agentic Data Pipeline replaces manual processing with AI agents that invoke specialized operators. A case study shows a financial logic problem where the agent writes Python code, reducing the answer error from 0.001 to 7.63×10⁻⁸, demonstrating automatic detection and removal of low‑quality samples.

The Post‑Training Data Production Platform is being upgraded with (1) a unified seed library, (2) framework integration for direct consumption of seed services, and (3) a consolidated OCTO gateway that aggregates synthesis channels and APIs.

Overall, the article argues that LLM data production must evolve from sheer volume collection to a verifiable, iterative quality‑controlled system that spans text, image, and audio modalities, leveraging agents, automated evaluation, and lake‑style storage.