How Momo Leverages Large Model Technology to Transform Business and R&D Processes

Introduction: Momo shares how it adopts large model technology to innovate its business and R&D workflows.

AI Application Paradigm Update: Traditional AI pipelines rely on models like BERT and YOLO for content safety, understanding, classification, and representation, producing logits, embeddings, and labels. Large models simplify this to a text‑in‑text‑out paradigm, but integration still requires converting model outputs back to programmatic forms.

Efficient Inference: Momo improves inference using techniques such as PagedAttention, FlashAttention, Continuous Batching, W8A8/FP8 quantization, and Prefix Caching. Quantization reduces computation by converting float32 to int8, while Prefix Caching stores common prompt prefixes in KV cache to avoid recomputation, though it increases memory usage.

Performance Evaluation: Experiments show that FP8 quantization incurs less than 1% MMLU loss, while int8 causes ~2% loss. Combining Smooth Quant W8A8 with Prefix Caching and memory‑extended caching doubles throughput under a 1.5 s latency bound.

Large Model Application Platform: Momo builds a workflow‑based platform instead of RAG, enabling rapid iteration of model‑driven features such as prompt prefill, fallback nodes, text‑to‑speech control, and structured JSON outputs. The platform integrates with internal RPC, storage, and messaging systems.

Future Outlook: Plans include rewriting the inference engine in C++ to reduce vLLM overhead, expanding KV‑Cache capacity via remote storage, adding multimodal support, and focusing on C‑side vector‑database use for recommendation and clustering.

Conclusion: Large models greatly enhance Momo’s ability to solve complex internal problems, improve inference efficiency, and drive innovative applications, with further optimization and multimodal expansion slated for continued impact.

Q&A highlights: Structured output uses vLLM; copy overhead is estimated from KV‑Cache size and bandwidth; L20 GPUs provide linear performance gains; W8A8 quantization yields minimal accuracy loss; Prefix Caching achieves up to 90% hit rate in chat scenarios.