OpenMLDB: A Production‑Grade Feature Platform for Consistent Online and Offline Machine Learning

In many machine‑learning scenarios—real‑time recommendation, risk control, fraud detection—low‑latency, accurate feature supply is essential. OpenMLDB is an open‑source machine‑learning database that provides a production‑grade feature platform with consistent online and offline behavior.

The AI engineering process faces challenges such as massive data processing time, the need for correct and efficient feature provision, and millisecond‑level computation for real‑time decisions. OpenMLDB addresses these by delivering real‑time feature computation, defining real‑time data and computation, and supporting use cases like credit‑card fraud detection within 20 ms.

OpenMLDB’s architecture includes a batch SQL engine built on Spark for offline development and a real‑time SQL engine—a distributed, high‑availability time‑series database—optimised for feature extraction. An execution‑plan generator guarantees consistency between the two engines, and all development is performed via SQL.

The platform supports the full lifecycle from offline feature extraction and model training to online inference, ensuring low latency, high concurrency, and high availability. It offers both in‑memory and disk storage engines, pre‑aggregation techniques, double‑linked‑list data structures, and other optimisations to achieve millisecond‑level processing.

OpenMLDB integrates with data‑ecosystem tools (Kafka, Pulsar, Flink, HDFS, Hive, etc.), machine‑learning frameworks (XGBoost, TensorFlow, PyTorch), and orchestration platforms (Airflow, DolphinScheduler). Recent releases added new SQL syntax, batch support for the online engine, auto‑feature engineering, a Go SDK, and roadmap items such as expanded SQL capabilities and stability enhancements.

A case study of Akulaku, a fintech company, shows 4 ms latency on 1 billion orders. The Q&A section covers consistency between offline and online data, future support for Hudi/Iceberg, separation of offline/online tasks, and extensibility via UDFs written in C++ (with Python support planned).