From Files to Lakehouse: Scaling AI Training Data to 100B+ Tokens with Lance
IQuest Research replaced file-based data management with a Lance-based lakehouse on Volcano Engine EMR, introducing a four-layer architecture with stable record IDs, columnar accumulation, versioned delivery, and bidirectional traceability to handle 50B+ records and enable reproducible training at hundred-billion token scale.
As AI training data scales to hundreds of billions of tokens, traditional file-and-directory management becomes a bottleneck. IQuest Research, which trained the 320B-parameter IQuest-Q1 model on ~29T tokens, faced five acute problems with its Parquet-on-object-storage pipeline: schema evolution forced full rewrites; new labels required deriving entirely new datasets; version tracking relied on fragile directory names; ad-hoc deliveries broke reproducibility; and experiments could not be stably bound to the exact data version used.
The core issue
Files and directories cannot support continuous processing, version management, and high-frequency delivery at this scale. The team partnered with Volcano Engine to rebuild the data foundation on EMR using Lance, an open-source columnar lakehouse format designed for multimodal AI.
"4+1" architecture: four layers around one boundary
All layers are Lance tables sharing the same table, version, and index capabilities:
Source layer (贴源层) : Raw multi-source data lands as Lance tables at full granularity, ingested via continuous Append.
Pool table layer (池表层) : Two stages of pool tables perform format validation, anomaly filtering, deep deduplication, labeling, and scoring. Stable results are tagged without rewriting raw data.
Delivery layer (数据交付层) : A unified delivery table aggregates vetted pool data, passes quality checks, and is frozen with a Tag. This becomes the immutable contract between data and training teams.
Consumption layer (消费层) : Training jobs read target columns in parallel via Ray, mixing data by label, quality score, and source. Each experiment records the exact table version and Tag.
Four key design pillars
Stable record IDs persist across the entire pipeline, enabling record-level joins even when underlying files are rewritten.
Columnar accumulation : New labels and scores are added as columns via targeted updates on the same table, avoiding full rewrites.
Separated change strategies : Pool tables evolve continuously; delivery tables are immutable once tagged, giving training a stable snapshot.
Tag- and read-aware version cleanup : Versions actively read or protected by a Tag are never garbage-collected.
Engineering enhancements for production scale
Volcano Engine contributed optimizations across four components:
Lance Core & Lance Spark: Column pruning and filter pushdown let training read only needed fields, eliminating full scans.
Ray integration: Tag-based snapshot reads and multi-worker parallelism allow concurrent experiments without interference.
Catalog: Central registration of tables, versions, Tags, and indexes with unified access control.
These changes make the open-source format production-ready at hundred-billion scale.
Outcomes: quality built-in, bidirectional traceability, data transparency
Delivery quality built-in : The unified delivery layer acts as a standard contract; most defects are caught at the pre-delivery validation gate.
End-to-end bidirectional traceability : Forward reconstruction from source to training version; reverse lookup from a bad sample to its root cause via stable record IDs.
White-box transparency : Catalog exposes data location, version, mixing ratios, and sampling counts to all roles, enabling self-service hypothesis testing.
The platform now stably supports 50B-scale pool and delivery tables and is evolving toward trillion-scale. The shift from managing files to managing data assets — centered on tables, versions, and records — offers a reproducible blueprint for any AI team scaling training data.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
