Punica System: Enhancing AI Inference Service Efficiency Through FaaS Architecture

This article introduces the Punica system, a comprehensive platform designed to improve the efficiency of AI inference services. The system addresses challenges in service development, testing, deployment, and maintenance by implementing a one-stop platform and autonomous operation mechanisms.

The content understanding inference service processes text, images, and videos from multiple sources like Baijiahao, Haokan, Quanmin, and live broadcasts, marking content tags for Baidu's Feed information stream and search services. With nearly a thousand inference services accumulated over time, managing existing services and efficiently onboarding new ones has become increasingly challenging.

The article outlines key problems in the current workflow: complex framework parameters and high configuration costs during development; expensive test data construction and insufficient test resources; lengthy learning curves for multiple PaaS platforms and complicated monitoring setup during deployment; and slow deployment times (20+ minutes) with low elasticity and poor resource utilization.

The solution focuses on two main aspects: one-stop platform and autonomous operation. The one-stop platform unifies multiple platforms, reduces user learning costs, provides recommended parameter configurations, enables rapid test resource allocation, offers automated testing tasks, supports quick small-scale verification, and automates monitoring and alerting for new services. It also lowers business resource costs by supporting both Python and high-performance C++ microservice frameworks, multiple GPU types (T4, P4, A10, A30, Kunlun), and model performance optimization through PaddlePaddle integration.

The autonomous operation component includes resource auto-scheduling (regular resource recovery, time-based resource reuse, and capacity expansion/contraction), self-healing inspection systems (instance-level and service-level health monitoring, fault root cause analysis), and reduced maintenance costs (PaaS App reduction by one order of magnitude, simplified cloud-native transformation).

The technical architecture separates business environments from basic environments, achieving high elasticity through FaaS (Function as a Service) principles. The system consists of four main parts: FaaS system (container framework, scheduling system, proxy gateway), platform frontend (unified service access, testing, deployment management), platform backend (OpenAPI support, service management), and service governance (self-healing inspection, capacity scaling).

Key technical implementations include: container framework with heartbeat mechanisms, deployment capabilities, intervention interfaces, and debug mechanisms; service scheduling with Resource-to-App and Replica-to-Node mapping; service discovery through a unified gateway with authentication and rate limiting; and strategy controllers for autonomous operations including service inspection, capacity scaling, resource recycling, and idle task support.

The system has achieved significant improvements: 100% increase in new service onboarding efficiency, 5x faster service scaling, and recovery of over 400 GPU cards. It has been successfully deployed for new business iterations and development.