How Ant Group’s Ragent Leverages Ray for Scalable AI Agents

Background

Ray, the open‑source distributed framework originally built for large‑model training at OpenAI, has been adopted by Ant Group since 2017. Ant contributed over 26% of Ray’s core code, becoming the second‑largest contributor worldwide, and now runs more than 1.5 million CPU cores in production while maintaining the Ray community in China.

Motivation

From 2018 to 2022 Ant built several Ray‑based engines—Geaflow for flow‑graph computation, Realtime and Mobius for streaming and ML training, and Mars for inference and scientific computing. The team also pioneered a Multi‑Tenant architecture that the Ray community only began to consider later. In the 2023‑2024 era of large models, Ant delivered a Unified AI Serving framework that integrates offline, online, inference, and deployment workloads for its massive core business.

Design & Implementation

The Ragent framework is organized around four essential modules for LLM‑based agents:

Profile : Defines the agent’s persona (e.g., a gentle travel assistant) and the roles it can play.

Memory : Consists of Knowledge (domain and prior knowledge) and Experience (historical dialogues, user queries, reasoning steps, and action outcomes) to help the agent learn from past interactions.

Planning : Decomposes complex tasks into manageable subtasks using algorithms such as Chain‑of‑Thought or Tree‑of‑Thought, similar to flow‑chart design in programming.

Action : Executes real‑world tasks based on the plan and memory. A key feature is Function Calling, which lets the model invoke external services or even control physical devices like robotic arms.

These four components constitute the core of a large‑language‑model agent capable of influencing the real world beyond pure text generation.