Accelerating Large Language Model Inference with the YiNian LLM Framework

The article introduces YiNian LLM, a large language model inference acceleration solution that focuses on reducing latency and improving throughput.

It explains the two‑step inference process of Transformer‑based LLMs—prefill (full model pass) and decoding (token‑by‑token generation)—and describes how KVCache stores intermediate results to avoid recomputation.

Memory consumption is analyzed, showing how prefill consumes a fixed amount of GPU memory while decoding can grow with token length, and how batch size and KVCache quantization affect overall GPU utilization.

The YiNian LLM framework adopts hand‑written models and custom operators instead of static computation graphs, enabling flexible optimization across NVIDIA, Intel, and emerging accelerator hardware.

Its scheduling layer implements continuous batching and PageAttention to dynamically adjust batch composition, use page‑level KVCache management, and route shared prefix tokens, thereby increasing effective batch size and reducing idle GPU time.

Future plans include expanding model support (e.g., Llama, Baichuan), advancing scheduling techniques such as speculative decoding, and developing hardware‑specific custom operators for CPUs and specialized AI chips.

The Q&A section compares YiNian LLM with CTR inference optimizations, discusses GPU idle gaps during decoding, benchmarks against TensorRT‑LLM on A800/A100, and evaluates multi‑stream versus asynchronous large‑batch strategies for recommendation‑grade models.