Inference Performance Optimization for AI Applications: Methods, Case Studies, and Future Directions

Shan Zhou, an algorithm expert at Ctrip, leads AI application performance optimization on CPU and GPU platforms, covering computer vision, natural language processing, machine translation, and speech processing.

With the rapid development of deep learning, AI services are being deployed across many production scenarios, improving efficiency and reducing costs. Ctrip's vacation AI team has integrated computer vision, NLP, machine translation, and speech technologies into various tourism services such as intelligent customer service and search ranking. Their internally developed machine translation also supports the company's internationalization.

Increasing model complexity raises resource consumption, leading to higher costs, longer inference latency, and slower training speed, which negatively affect user experience and production efficiency. Optimizing both training and inference performance is therefore critical.

1. Background and Current State of Inference Optimization

Inference performance is a key factor for the success of AI products. It is influenced by hardware configuration, deployment methods, algorithm and model complexity, and deep learning frameworks.

1.1 Inference Service Performance Metrics

Latency (average, p90, p95, p99) and throughput (QPS/TPS or CPS) are the primary service metrics. Online services are latency‑sensitive, while offline services focus on high throughput. Additional quality metrics such as precision, recall, and task‑specific scores (e.g., BLEU for translation) are also important.

1.2 Main Deep Learning Frameworks

Common frameworks that support both training and inference include TensorFlow, PyTorch, and MXNet. Dedicated inference frameworks such as TensorRT, ONNX Runtime, OpenVINO, and PaddlePaddle are used for higher performance.

2. General Optimization Methodology

Optimization follows a systematic workflow: define performance goals, perform framework‑specific profiling, locate bottlenecks, devise and evaluate strategies, implement optimizations, test results, and iterate until targets are met.

2.1 Optimization Flow

The flow is illustrated in Figure 2 (not reproduced) and consists of goal setting, analysis, bottleneck identification, strategy formulation, execution, testing, and iterative refinement.

2.2 Optimization Techniques

Optimizations are divided into system‑level (code, runtime, hardware‑specific) and model‑level (algorithmic) methods. System‑level techniques include SIMD acceleration, MKL‑DNN, cuDNN, and other hardware‑specific libraries. Model‑level techniques encompass quantization, pruning, low‑rank approximation, and knowledge distillation, often combined with system‑level improvements.

3. Practical Case Studies

Case studies focus on a Transformer translation model (Encoder‑Decoder). Profiling with TensorFlow’s timeline revealed that matrix multiplication (self‑attention, cross‑attention, FFN) dominates runtime, CPU utilization is low (~40 %), and many redundant transpose operations exist.

Optimization actions included:

Using high‑performance math libraries (MKL, cuDNN) for matrix multiplication.

Re‑designing memory layout to eliminate unnecessary transposes.

Micro‑architectural tuning for better parallelism.

Operator fusion to reduce kernel launch overhead.

After applying operator fusion, memory layout redesign, and float‑16 precision on a T4 GPU, latency decreased significantly across various batch sizes (Figure 7) and throughput improved with longer token sequences (Figure 8). Similar gains were observed on CPU platforms for BERT, ALBERT, and other models (Figure 9).

4. Future Development and Outlook

As AI models grow in depth and width, inference demands will only increase. Manual per‑model optimization is labor‑intensive; automated compilation techniques like TVM are expected to handle the majority of optimizations, leaving only the final 20 % for expert fine‑tuning.

Continued advances in compilation‑based optimization will lower deployment costs, accelerate AI adoption in the tourism industry, and enable higher‑quality services for customers.