How Ant Group’s Ray‑Powered Ragent Redefines LLM‑Based AI Agents

Background

Ray, the open‑source distributed framework originally built for large‑model training at OpenAI, has been adopted by Ant Group since its early days. Ant contributed over 26% of Ray’s core code, becoming the second‑largest contributor worldwide, and now runs more than 1.5 million CPU cores in production while maintaining the Ray community in China.

Since 2017 Ant’s Ray team launched the first business‑scene flow‑graph engine Geaflow in 2018. Between 2018 and 2022, leveraging Ray they built several engines such as Realtime, the open‑source Mobius engine, and the scientific‑computing engine Mars. They also pioneered a Multi‑Tenant architecture, which the Ray community only began to consider later.

In the 2023‑2024 era of large models, Ant delivered Unified AI Serving, a framework that unifies offline, online, AI inference, and AI deployment, serving as a core scenario for their 1.5 million‑core workload.

Motivation

To support the growing demand for large‑language‑model (LLM) agents in finance and other domains, Ant needed a scalable, Ray‑based framework that could orchestrate complex agent workflows, manage state, and integrate with external functions.

Design & Implementation

The Ragent framework is organized around four core modules that together constitute a functional LLM‑based agent:

Profile : Defines the agent’s persona and role, e.g., a gentle travel assistant that can manage itineraries and perform data analysis.

Memory : Split into Knowledge (domain and prior knowledge) and Experience (recorded dialogues, user queries, reasoning steps, and action outcomes) to enable continual learning and error avoidance.

Planning : Decomposes complex tasks into manageable subtasks using algorithms such as Chain‑of‑Thought or Tree‑of‑Thought, akin to flowcharts in software design.

Action : Executes tasks based on experience and plans. A key feature is Function Calling, allowing the model to invoke external services or even interact with physical devices like robotic arms.

These four modules form the essential building blocks of an LLM‑based agent in Ant’s Ragent framework.