OmniTable: One Logical Table Manages 305B Records Across 35PB for LLM Training

Ant Group's OmniTable is a unified wide-table system for petabyte-scale LLM data curation and exploration, recognized with the VLDB 2026 Industrial Track Best Paper award. The system manages over 35 PB and 305 billion records across Web, code, PDF, and SFT domains in production.

Problem: Fragmented Physical Tables and Operational Overhead

Traditional LLM data pipelines create a new physical table for each processing stage (parsing, cleaning, quality scoring, deduplication, safety labeling) and each data domain. As sources and features grow, engineers face three core costs:

Data hard to locate – hundreds of tables, no stable logical view.

Features hard to backfill – adding one feature required handling 106 tables in a real case.

Results hard to trace – UDFs scattered across codebases, missing column-level lineage between input batches, operator versions, and downstream training jobs.

Solution: Logical Unification, Physical Separation

OmniTable adopts a "logical unification, physical separation" principle. At the logical layer, each row is a traceable data entity identified by a global _ai_unique_id_, and each column represents a processing state or derived feature (RawData, ProcessedData, TrainableData, quality, domain, safety, deduplication, etc.). A second system field _ai_append_name_ records ingestion batch, source, and version.

Four domain logical wide tables (Web, Code, PDF, Post-SFT) collectively manage 35+ PB and 305B+ records. The largest Web table holds ~25 PB, 300B+ records, 800+ logical columns, and 200+ registered features. A controlled experiment on ~2 PB data extended logical columns to 2,500 .

A Catalog maintains the mapping from logical columns to physical locations (row splits, column splits, merged small files, partition adjustments, materialized views for hot columns). Physical reorganization adds ~8–15% storage overhead for materialized hot-column groups but leaves upper-layer schemas unchanged.

Feature Computation: From Task Orchestration to Target-Column Declaration

Features are registered with input columns, output column, UDF/SQL/model inference logic, version, and CPU/GPU execution preference. Engineers submit a backfill job by specifying target batch and target feature. OmniTable:

Queries current column-level computation state.

Traverses the column-level dependency DAG to find the minimal unfinished dependency closure.

Generates a physical execution plan in topological order, reusing already-computed results.

Merges operators that share inputs and run on the same engine into a single scan.

On success, the Catalog atomically records batch–feature status, version, physical location, and column-level lineage. Uncommitted results never enter the stable logical view.

Record-Level Fault Tolerance

Non-structured corpora contain anomalous encodings, oversized texts, or corrupted content. Traditional batch jobs fail the entire task on a single UDF OOM or timeout. OmniTable isolates common UDF failures to the record level:

Each UDF call carries timeout and memory checks.

On Python OOM, timeout, or uncaught exception, the system logs sample ID, exception type, and error summary, writes NULL for that record, and continues processing others.

Error records go to an error table for later repair and recomputation.

In a 500 GB, ~600M-record feature task with 31,247 anomalies (0.005%) , enabling failover completed 99.995% of records in 6.2 hours with zero manual intervention . Without it, the job failed and required three rounds of investigation, deletion, and resubmission totaling ~52 hours (18 hours manual). Record-level wrapping adds ~3–5% execution overhead.

Operator Fusion and Adaptive Tuning

LLM feature computation varies: text length, char ratios, rule filtering suit CPU/SQL; model inference may run on CPU or GPU. OmniTable routes operators to Spark, MaxCompute SQL, or GPU inference platforms based on user declaration, operator profiles, engine capabilities, and cluster load, and adjusts resource parameters using historical runs.

Operator fusion merges multiple features reading the same column on the same engine into one scan. In an experiment with 8 CPU/Spark features reading parsed_text on ~2.5 PB / 300B+ records:

Scans reduced from 8 to 1 .

CPU hours dropped from 42,000 to 18,500 (55.9% reduction) .

End-to-end time fell from 38 hours to 14 hours (2.7× speedup) .

Adaptive tuning on 50 GB, 500 GB, and 2 TB batches for a specific BERT feature task achieved 100% first-submit success rate with task cost within 5% of expert manual tuning . This demonstrates near-expert configuration for that workload, not universal optimality.

Background Governance: Continuous Physical Layout Evolution

As the wide table grows (new batches, new features), small files accumulate, partitions skew, column counts rise, and query hotspots shift. A background governance service continuously observes these metrics and automatically executes:

Small-file compaction

Row splitting

Column splitting (allows logical schema to exceed underlying engine's physical column limit ~1,200)

Materialized view construction for hot column groups

Governance follows a Prepare–Execute–Commit protocol: prepare new layout, verify, then atomically switch Catalog mapping. Old layout serves queries until cutover; failed governance rolls back. Users query the same logical columns unaware of physical changes.

In a fixed ~2 PB test with unchanged query column sets, growing logical columns from 200 to 2,500 increased P95 latency from ~25 s to ~38 s, crossing the engine's ~1,200-column physical limit. A scale test from 1 TB to 25 PB maintained filter-export throughput at 18–23 TB/hour.

Single-sample lookup uses a global ID index mapping ai_unique_id to physical table, partition, and row group. On the 25 PB / 300B+ record / 800+ column Web dataset, full logical row query P50 = 8.3 s , P99 = 14.7 s ; full scans took 184 s and 612+ s respectively.

Multi-column exports leverage background materialized views to pre-join hot column groups. A representative scenario joining 15 columns across 4 physical tables improved filter-export throughput from 4.8 TB/hour to 20.1 TB/hour .

End-to-End Evaluation: Real SFT Data Preparation Task

A production SFT data preparation task with 8 data sources and 12 features (9 CPU UDFs, 3 GPU inferences) was compared:

Data ingestion & access : Old Pipeline ~2 days, OmniTable ~0.5 days

Feature backfill : Old Pipeline ~9.5 days, OmniTable ~1.7 days

Multi-table join & export : Old Pipeline ~2.5 days, OmniTable ~0.3 days

Total cycle : Old Pipeline ~14 days, OmniTable ~2.5 days

Manual steps : Old Pipeline 45, OmniTable 12

Independent pipelines/scripts : Old Pipeline 24, OmniTable 10

Physical tables used : Old Pipeline 35, OmniTable 1 logical wide table

End-to-end speedup: 5.6×; manual steps reduced 73.3%; independent pipelines/scripts reduced 58.3%. The gains come from eliminating per-table orchestration, avoiding repeated scans of shared inputs, preventing rare anomalies from triggering full-batch reruns, and deferring physical layout tuning to automated background governance.

Trade-offs and Applicability

The approach incurs explicit costs: extra storage for materialized hot columns (8–15%), modest execution overhead for record-level fault tolerance (3–5%), and cluster resources for background governance. Different organizations have varying data domains, compute engines, and team practices, so OmniTable serves as a reference system design validated at 35+ PB production scale . Its core insight: stabilize logical semantics first, then let physical layout evolve continuously .

As LLM training enters the petabyte era, the challenge shifts from running a single job to keeping continuously growing data, features, and compute manageable over the long term. OmniTable aims to save not only machine time but also engineer time spent repeatedly locating tables, patching jobs, and debugging anomalies.

Paper: OmniTable: A Unified Wide-Table System for Petabyte-Scale LLM Data Curation and Exploration , PVLDB Vol. 19, No. 12, pp. 4276–4289, DOI: 10.14778/3827998.3828032. PDF: https://www.vldb.org/pvldb/vol19/p4276-fu.pdf