How FaaS Transforms AI Platforms: Lessons from Haro’s Cloud‑Native Journey

Why the AI Platform Needs FaaS

The AI platform faces three major pain points: complex operations due to heterogeneous model inference services (Python, C++, Java, etc.) and varied deployment patterns; stability issues caused by hotspot models and slow auto‑scaling under burst traffic; and high IDC costs stemming from low resource utilization.

Current Pain Points of the AI Platform

Operational complexity: managing hundreds of models across multiple languages and container configurations.

Stability: centralized deployment creates hotspots and resource contention during traffic spikes.

Cost inefficiency: low utilization of IDC resources leaves significant optimization space.

Requirements for the New Architecture

The platform is divided into an online service domain (decision, feature) and a model training domain (model, training). Desired capabilities include rapid response to burst traffic, zero‑scale for low‑frequency models, easy A/B testing for fast‑iteration models, controllable costs, and simplified operations and deployment.

Cloud‑Native Evolution and FaaS Selection

Moving from traditional Kubernetes clusters to a serverless FaaS model promises extreme elasticity, the ability to scale to zero, reduced operational overhead, and better alignment with AI workloads that are stateless, short‑lived, and have unpredictable traffic patterns.

Why Knative Was Chosen

After evaluating several options, Knative was selected because it supports multiple triggers (Eventing, HTTP, gRPC), offers zero‑scale elasticity, and provides version management and traffic‑splitting features that are valuable for AI model deployments.

Practical Implementation of FaaS in the Model Platform

FaaS adoption brings four key benefits: upgraded platform capabilities (support for large and GPU models), improved stability through model isolation, increased engineering efficiency via self‑service model publishing, and reduced IDC costs by lowering online service expenses.

FaaS Deployment Process

Engineers can upload models, define input/output schemas, and select a one‑click FaaS deployment option. The system routes requests to the appropriate FaaS cluster, abstracting heterogeneous back‑ends (Python, GPU, PMML, TensorFlow) and dramatically lowering operational effort.

Automatic Load Testing & Standardized Specs

By integrating with a load‑testing platform, the team automatically evaluates pod resources and standardizes specifications before invoking native cloud‑native FaaS APIs for deployment, ensuring optimal resource allocation and scaling behavior.

Cold‑Start Optimization

To mitigate cold‑start latency, a model distribution service pre‑downloads model assets, reducing startup time from ~150 ms to ~10 ms for a single model.

Graceful Model Warm‑Up

For large deep‑learning models, the team leverages Knative’s versioning, traffic routing, and blue‑green deployment capabilities together with a custom GraySDK to provide smooth warm‑up and avoid sudden latency spikes.

Case Study: Haro Smart Scheduling FaaS Migration

The smart‑scheduling service, a core scenario for two‑wheel logistics, processes massive, city‑specific models with highly variable load. By converting timing prediction, feature extraction, and model inference to FaaS, IDC costs dropped 35 % and overall performance improved 20 %. Serverless’s zero‑maintenance, strong isolation, and pay‑per‑use characteristics also eliminated idle resources and enabled rapid scaling during peak periods.

Future Outlook of FaaS in AI Platforms

Beyond the model platform, FaaS is planned for feature services (handling hot‑cold feature distribution), internal admin back‑ends (sporadic usage), and scheduled prediction tasks (burst traffic). Additional business domains such as intelligent customer service chatbots, promotional marketing spikes, and IoT sensor processing are also strong candidates for serverless adoption.