Tencent CodeBuddy’s AI DLC Slashes Training Time and Costs with a Unified Spark‑Ray Service

In traditional AI pipelines, Spark is used for batch processing while Ray handles model training, requiring two separate clusters. This setup forces repeated data transfers, duplicate permission configurations, and leaves up to 80% of GPU capacity idle while waiting for data loading.

To address these inefficiencies, the Tencent CodeBuddy team built an integrated platform called AI DLC (Data Lake Compute). It is the first production‑grade, fully managed service that combines Spark and Ray, allowing data to remain in place from preprocessing through model serving. Each interaction generated by an agent is automatically persisted as a training resource for subsequent iterations, turning what would be waste into valuable fuel.

According to the team’s reported results, the unified approach compresses the training iteration cycle from a month‑scale to a week‑scale. It also supports in‑place computation on feature datasets at the billion‑record level and reduces operational expenses by roughly 60%.

The platform was publicly presented at a Tencent Cloud big‑data event in Shenzhen on July 25, where the team shared practical challenges and implementation details of building an AI‑native data foundation.