Tencent FinTech AI Development Platform: Architecture, Challenges, and Solutions

The Tencent FinTech AI development platform was built to support four core financial services—mobile payments, investment wealth management, livelihood services, and cross‑border payments—by providing a one‑stop development environment that improves feature engineering, model development efficiency, training capability, and inference stability.

The platform evolved through four stages: traditional machine learning, deep learning, a unified feature platform (2022), and a unified inference platform (2022). Despite these advances, developers still faced low efficiency and high entry barriers, prompting the construction of a comprehensive AI development workflow.

Key technical challenges identified in 2022 included (1) feature‑engine performance and quality, (2) fragmented model development leading to duplicated effort, (3) scaling training for large‑scale samples and parameters, and (4) ensuring stable, high‑throughput inference services.

Solutions implemented:

Feature engine architecture with separate online and offline services, supporting feature selection, sample rollback, online feature publishing, and continuous monitoring.

Feature selection pipelines using quality metrics and importance‑based algorithms (filter, wrapper, embedded methods), plus a continuous‑learning CTR model that jointly trains feature interactions.

Sample rollback optimizations using business‑level feature partitioning, sparse storage, Bloom filters for join acceleration, dictionary‑based public features, and hybrid in‑memory/disk joins.

Feature online strategies that split by update frequency, store cold data in HBase and hot data in CKV, and employ read‑write separation.

Training optimizations involved upgrading to TensorFlow 2, using TFRecord with balanced file counts, GPU pre‑loading of upcoming batches, sparse‑embedding kernel acceleration, mixed‑precision computation, multi‑card training with Horovod, model‑parallel for sparse layers, data‑parallel for dense layers, multi‑stage pipelines, and a three‑level cache (SSD, memory, GPU) for embedding reads.

Deployment enhancements introduced a unified inference service with a visual UI for model rollout, verification, and traffic switching; model‑switch validation with rollback fallback; inference acceleration via operator optimization, model pruning, and quantization; and cloud‑native service governance for disaster recovery, fault tolerance, and elastic scaling.

Stability measures included strict change‑management processes, code and dependency performance tuning, adherence to development standards, and regular disaster‑recovery drills, aiming for rapid detection (1 min),定位 (5 min), and recovery (10 min) of incidents.

The platform also provides full‑link operation monitoring covering feature/sample health, training metrics (e.g., AUC), and online inference metrics (error codes, latency, effectiveness), integrated with AB‑testing systems.

Future plans focus on expanding large‑scale graph training capabilities and further automating hyper‑parameter tuning and model selection through AutoML techniques.

Q&A addressed openness (the platform is not open‑source due to internal dependencies), Bloom filter usage, key factors for inference stability, and the impact of large models on architecture and performance.