Daft, Ray, and Lance: Rebuilding AI Data Infrastructure from Compute to Storage

Hello everyone, we meet again. Starting today we begin a special series on Daft, Ray, and Lance. Regular readers may have noticed our update frequency has not been high this year, largely because the technology stack in AI and Data directions is changing rapidly.

How should AI-era data infrastructure recombine compute, scheduling, and storage?

Today we provide a basic understanding and cognitive upgrade: what are the positioning and roles of these three frameworks in the current AI era?

Daft

Daft is an open-source high-performance data compute engine from Eventual, designed for AI and multimodal scenarios. It offers a Python DataFrame API while building core execution in Rust. Its goal is not to replace general big data platforms like Spark or Flink, but to solve increasingly prominent problems in AI data processing: fragmented multimodal data tools, insufficient Python data pipeline performance, and the lack of a unified data processing abstraction from raw data to model input.

Unlike traditional DataFrames that mainly handle structured tables, Daft focuses on AI data types such as images, text, vectors, and tensors. It attempts to unify data reading, cleaning, transformation, model invocation, and result write-back into a single DataFrame pipeline. This allows workflows that previously required stitching together Pandas, PIL, NumPy, Ray Data, and other tools to be expressed and optimized within one compute framework.

From the perspective of the entire AI data infrastructure, Daft sits in the middle as an "AI data compute layer." Downward, it connects to object stores like S3 and GCS, as well as lakehouse formats and systems such as Iceberg, Delta Lake, Hudi, Parquet, and Lance. At the runtime layer, it can execute locally or leverage Ray for distributed compute. Upward, it connects to training, inference, embedding, LLM, and evaluation model tasks.

Therefore, Daft's core value is not building another SQL engine, but connecting "data engineering" with "AI engineering": enabling structured data, multimodal data, and model compute to collaborate in the same data pipeline. Simply put: Daft is to AI what Spark is to traditional data processing.

Ray

Apache Ray is an open-source distributed compute framework for AI and Python workloads. Its core goal is not to define how data should be processed, but to solve how compute tasks can be efficiently scheduled and executed on a single machine or a cluster.

Ray provides concise distributed programming abstractions: Task, Actor, and Object Store. Developers can turn ordinary Python functions into distributed tasks, use Actors to manage stateful long-running compute instances, and share data across processes and nodes via a unified object store. Meanwhile, Ray perceives CPU, GPU, and custom resource demands, scheduling different task types to appropriate nodes.

These capabilities suit AI scenarios well. Today's AI pipelines are rarely single-type compute: document parsing may be CPU-bound, image and video processing may need both CPU and I/O, embedding and model inference require GPUs, while data I/O accesses S3, object stores, or remote services. Ray's value lies in unifying these heterogeneous tasks into a single distributed runtime for management and scheduling.

In the AI data infrastructure stack, Ray acts as the underlying "distributed compute runtime." Downward, it manages machines, CPUs, GPUs, memory, and cluster resources. Upward, it hosts frameworks like Daft, Ray Data, Ray Train, and Ray Serve for data processing, training, and serving. For Daft, Daft defines DataFrames, query plans, and data processing logic, while Ray distributes that compute to cluster workers.

Thus, Ray can be simply understood as: it does not decide how AI data "should be computed," but decides where these computations run, with how many resources, and how they run in parallel across the cluster. Is it like Apache Yarn's role? Certainly Ray has many more modules and functionality far beyond Yarn.

Lance

Lance is an open lakehouse format designed for AI and multimodal scenarios. Its core goal is not to replace traditional databases, but to solve the long-standing disconnect between AI data storage, random access, vector search, and model training. Unlike traditional columnar formats oriented toward analytical scans, Lance emphasizes efficient random access on object storage, vector and full-text search, and the high-selectivity read capabilities required for ML training.

Lance is more than a single file format. It now encompasses File Format, Table Format, Index, and Catalog layers: the bottom layer handles data storage, while upper layers handle table versioning, schema evolution, ACID transactions, and vector, full-text, and scalar indexing capabilities. Lance Tables organize data using versioned Fragments, Manifests, and Indexes, supporting ACID, Time Travel, and incremental updates, making them suitable both as AI dataset storage and as direct participants in retrieval and feature engineering.

This design fits modern AI pipelines. A dataset often contains text, images, audio, video, metadata, and embeddings simultaneously. Model training needs random sample reads, while RAG needs to quickly return to raw data based on vector search results. Built on the Apache Arrow type system, Lance naturally stores vectors, arrays, and other complex types, integrating Vector Search, Full-text Search, and Scalar Index as part of the data format system.

In the AI data infrastructure, Lance serves as the bottom "AI data storage and retrieval layer." Downward, it can be built directly on S3, GCS, and other object stores. Upward, it can be accessed by compute and training frameworks such as Daft, Ray, PyTorch, DuckDB, and Spark. It solves not "how compute tasks execute," but keeps multimodal raw data, embeddings, indexes, and version management in the same dataset, reducing repeated data copying between data lakes, vector databases, and training storage.

Therefore, Lance can be simply understood as: it handles how AI data "is stored, how it is read fast, how it is searched"; Daft handles how data "is computed"; Ray handles where those computations "run."

We now have a basic grasp of the division of labor and roles of these three frameworks in the AI Data era. We will expand on each framework in subsequent articles.