Tagged articles
inference acceleration
41 articles · Page 1 of 1
Tencent's Hunyuan Hy4 Flash model reduces average per-task cost to 1/20th of Hy4 and 1/13th of Hy4 preview on internal WorkIndex benchmarks, while achieving satisfaction scores near Claude Haiku 5.5 at a fraction of the cost through mixed attention architecture, distillation, RL token reduction, and 4-bit inference acceleration.
Claude Haiku 5.5Hy4 FlashLLM cost optimization
0 likes · 7 min read

Alibaba's Qwen-Audio-Jev integrates Jev decision heads into an audio LLM, enabling direct classification, binary judgment, and scoring without autoregressive decoding, delivering ~20x inference speedup across 18 benchmarks including speaker verification (6.5% EER) and audio quality assessment.
Jev decision headsQwen-Audio-JevRLCD
0 likes · 14 min read

Apple and CMU researchers introduce SSR, a framework that replaces free-form reasoning in multimodal search agents with selection from six predefined reasoning options, cutting per-step latency over 90% while matching task success rates across seven benchmarks.
Multimodal AgentsQwen3-VLRL without SFT
0 likes · 12 min read

VideoJev replaces autoregressive answer generation with a unified candidate-scoring interface for temporal grounding and multiple-choice QA, achieving 100-129x speedup over thinking-mode baselines while matching accuracy, with open-source 4B and 35B models, though calibration and false-positive issues remain.
Multimodal ModelsVideoJevcandidate scoring
0 likes · 5 min read

LingBot-VA 2.0 presents an industry‑first embodied‑native pre‑training model that aligns video prediction with action generation, uses a semantic‑visual‑action tokenizer, multi‑chunk prediction, foresight reasoning, and a sparse MoE architecture to achieve higher success rates and up to 6.5× faster end‑to‑end inference on real‑world robot tasks.
Embodied AIForesight ReasoningMulti‑Chunk Prediction
0 likes · 17 min read

The article compares DSpark and JetSpec, two recent open‑source speculative decoding frameworks that tackle inference efficiency from system‑level verification reduction and algorithmic token‑acceptance improvements, respectively, showing up to 9.64× end‑to‑end speedup on Qwen3‑8B and significant gains across math, code, and dialogue benchmarks.
DSparkJetSpeccausal consistency
0 likes · 14 min read


Geek LabsJun 29, 2026 · Artificial Intelligence
DeepSpec, an open‑source framework from DeepSeek, accelerates large‑language‑model inference by 2–5× through speculative decoding, where a lightweight draft model generates candidate tokens that the target model validates in parallel, reducing the serial bottleneck of autoregressive decoding and offering a full‑stack pipeline from data preparation to evaluation.
DeepSpecGPUPython
0 likes · 6 min read

The article details a systematic practice—starting from the “iceberg” challenges of finance, through data and knowledge engineering, reverse knowledge extraction with REER, multi‑dimensional synthetic data generation, prompt engineering (APO), cost‑aware fine‑tuning, inference acceleration, and emotion‑value evaluation—culminating in actionable guidelines for deploying large models in banking scenarios.
Knowledge EngineeringPrompt Optimizationemotion evaluation
0 likes · 14 min read

The Latent‑Condensed Attention (LCA) method dramatically cuts KV‑cache memory by 90%, speeds up pre‑fill by 2.5× and reduces decode latency by 1.8× for 128K‑token contexts, while requiring no extra parameters and preserving model performance across diverse LLMs.
KV cache reductionLCATransformer
0 likes · 10 min read

The DFlash approach replaces speculative decoding’s autoregressive drafter with a block diffusion model and injects target‑model hidden features into every KV‑cache layer, achieving up to 5× speed‑up for Qwen3.5‑27B on single‑GPU and 1.5–1.9× on high‑concurrency workloads while preserving output quality.
DFlashQwen3.5SGLang
0 likes · 12 min read

TC‑Padé uses a Padé‑based residual prediction framework, step‑aware strategies, and a trajectory‑stability indicator to accelerate diffusion sampling to as few as 20 steps while preserving visual fidelity, achieving up to 2.88× speed‑up on image generation and 1.72× on video generation.
Image GenerationPadé approximationTC-Padé
0 likes · 12 min read

The SSD framework and its SAGUARO optimization, developed by researchers from Stanford, Princeton, and Together AI, parallelize drafting and verification in speculative decoding, eliminating serial dependencies and achieving up to 2× faster inference than the world’s strongest engines and up to 5× speedup over standard autoregressive generation, while addressing challenges such as prediction accuracy, acceptance‑rate trade‑offs, and fallback strategies.
SAGUAROSSDinference acceleration
0 likes · 7 min read

Google Research’s TurboQuant algorithm compresses large‑language‑model KV caches from 32‑bit to 3‑bit, achieving a six‑fold reduction in memory usage and an eight‑fold inference speedup on H100 GPUs while preserving 100 % accuracy, and it also improves vector search performance without requiring large codebooks.
AI EfficiencyLLM compressionTurboQuant
0 likes · 10 min read

Taalas embeds the Llama 3.1 8B model directly into a 6nm ASIC, delivering 17,000 tokens per second—nearly ten times faster than top NVIDIA GPUs—while cutting system cost by over tenfold and power consumption by tenfold, albeit with limited flexibility and quantization trade‑offs.
AI hardwareASICLlama 3.1
0 likes · 10 min read

The article outlines the four major obstacles hindering industry‑specific LLM agents—manual workflow maintenance, poor knowledge reuse, training‑inference inefficiency, and complex reasoning evaluation—and explains how Huawei Noah’s MindScale package tackles each with self‑evolving workflows, automated prompt optimization, and a novel KV‑Embedding cache that slashes token consumption by 5.7× while boosting inference speed up to 70%.
Industry AgentKV-EmbeddingMindScale
0 likes · 7 min read

The Tencent WeChat AI team introduces WeDLM, a diffusion language model that, through topological reordering, surpasses autoregressive models on the industrial‑grade vLLM engine with over threefold speedup on math reasoning and up to tenfold in low‑entropy scenarios, and provides a step‑by‑step online tutorial with GPU compute credits.
GPU computeTencent AIWeDLM
0 likes · 5 min read

The article analyzes Google’s shift from scaling‑law to efficiency‑law, detailing how speculative decoding, language‑model cascades, distillation, CALM, accurate quantized training, and the Mixture‑of‑Recursions architecture together form a multi‑layered strategy to cut inference cost, boost throughput, and sustain the company’s AI moat.
Google TPULanguage Model Cascadesinference acceleration
0 likes · 8 min read

This article examines the challenges of optimizing video generation model inference, moving from operator-level tweaks to full-graph compilation using torch.compile, and details systematic strategies to eliminate Graph Breaks, handle dynamic shapes, KV-Cache indexing, and Python-side caches, achieving a 47.6% speedup on a 14B model without accuracy loss.
AIgraph optimizationinference acceleration
0 likes · 14 min read

Render‑of‑Thought (RoT) proposes a novel visual‑latent reasoning framework that compresses textual chain‑of‑thought into dense image embeddings, achieving faster inference, better interpretability, and plug‑and‑play integration without costly pre‑training, as demonstrated on multiple math and logic benchmarks.
Chain-of-ThoughtImplicit CoTLLM
0 likes · 11 min read

Jeff Dean highlighted speculative decoding as a lossless inference acceleration technique that can boost large language model throughput by 2–3×, and the article breaks down its core concepts—including parallel token verification, draft‑target model collaboration, rejection sampling theory, and practical optimizations such as continuous batching and tree‑based verification.
Draft-Target ModelKV CacheRejection Sampling
0 likes · 8 min read

This article explains why Transformer models dominate modern AI agents, outlines the challenges of large parameter counts and latency, and presents a comprehensive guide to model compression (parameter sharing, knowledge distillation, quantization, pruning) and inference acceleration (parallel computing, optimized attention, TensorRT deployment), complete with PyTorch code examples and a real‑world case study showing speed‑up and storage savings.
AI agentPyTorchTransformer
0 likes · 34 min read

SpecExit combines early‑exit and speculative decoding to let large reasoning models detect when they have almost finished thinking, trimming redundant chain‑of‑thought steps, reducing over‑thinking by 72% and achieving up to 2.5× faster end‑to‑end inference without noticeable accuracy loss.
AIearly exitinference acceleration
0 likes · 6 min read

The article analyzes a recent Duke University paper that reveals a "scratchpad" mechanism in diffusion large language models, proposes the DPad method to prune redundant suffix tokens before decoding, and demonstrates up to 61.4× faster inference with unchanged or even improved accuracy across multiple benchmarks.
DPadDiffusion LLMinference acceleration
0 likes · 10 min read

SpecForge, an open‑source training framework built on Eagle3, enables end‑to‑end speculative sampling for ultra‑large language models, integrates tightly with the SGLang inference engine, offers online and offline training modes, supports advanced parallelism strategies, and demonstrates up to 2.18× inference speedup on benchmark tests, with all code and pretrained drafts available on GitHub and Hugging Face.
AI performanceinference accelerationopen-source
0 likes · 9 min read


AIWalkerApr 28, 2025 · Artificial Intelligence
SimpleAR is a minimalist autoregressive visual generation framework that, with only 0.5 B parameters, achieves competitive 1024×1024 image synthesis through a three‑stage pipeline of large‑scale pretraining, supervised fine‑tuning, and GRPO‑based reinforcement learning, and demonstrates significant inference speedups using KV‑cache, vLLM, and speculative decoding.
Pretrainingautoregressive generationbenchmark
0 likes · 14 min read

This article examines the current state and future prospects of large AI models and inference acceleration, covering technology trends, diverse application scenarios from research to industry, and the challenges and opportunities that lie ahead for intelligent data governance, multimodal agents, and AGI.
AGIAIdata governance
0 likes · 11 min read

The article outlines how exploding LLM sizes create compute, memory, and latency bottlenecks and proposes a full‑stack solution—operator fusion, high‑performance libraries, quantization, speculative decoding, sharding, contiguous batching, PageAttention, and specialized frameworks like MindIE‑LLM—to dramatically boost inference throughput and reduce latency, while highlighting future ultra‑low‑bit and heterogeneous hardware directions.
Multi-modalOperator Fusioncontinuous batching
0 likes · 21 min read

This article reviews the evolution of Multi‑Token Prediction (MTP) techniques—from early blockwise parallel decoding to Meta's and DeepSeek's implementations—explaining their architectures, training and inference workflows, and the speed‑up gains they offer for large language models.
DeepSeekLLMMTP
0 likes · 20 min read

The article details the design and deployment of 360's AI Compute Center, covering GPU server selection, high‑performance networking, Kubernetes‑based cluster management, advanced scheduling, training and inference acceleration techniques, and a comprehensive AI development platform with visualization and fault‑tolerance features.
AI infrastructureGPU clusterKubernetes
0 likes · 21 min read

HASS aligns training and decoding contexts and objectives for speculative sampling, using harmonized objective distillation and multi-step context alignment, achieving 2.81–4.05× speedup and 8%–20% improvement over EAGLE‑2 while preserving generation quality in real-world deployments at Xiaohongshu.
AIHASSinference acceleration
0 likes · 11 min read

This article details how Alibaba's Wanxiang Lab adapted the SDXL diffusion model for large‑scale e‑commerce image generation, introducing the EcomXL series, a weighted‑distillation fine‑tuning method, hierarchical model fusion, specialized ControlNet variants, and the SLAM inference accelerator to achieve high‑quality, controllable product images within three seconds while boosting business metrics.
AIGCControlNetEcomXL
0 likes · 14 min read

EcomXL enhances SDXL for large‑scale e‑commerce image generation by leveraging tens of millions of curated images, a two‑stage fine‑tuning with denoising‑weighted distillation and layer‑wise fusion, specialized ControlNets for inpainting and soft‑edge consistency, and the SLAM inference accelerator to achieve sub‑second generation while boosting visual quality and adoption metrics.
AIGCControlNetEcomXL
0 likes · 15 min read

Deploying a dedicated inference cluster and applying four key optimizations—FlashAttention‑based attention computation, PageAttention KV‑cache management, Mixture‑of‑Experts parameter reduction, and tensor parallelism—can accelerate large language model inference by up to 50% for models as large as 70 B parameters while cutting deployment costs.
FlashAttentionPageAttentioninference acceleration
0 likes · 17 min read

This talk explores how large‑model AI can give game NPCs personality, outlines the opportunities and challenges of intelligent NPCs, presents a case study of the "Jue Zhi An Nuan" NPC, and discusses future directions, safety compliance, and real‑time multimodal interaction solutions.
AIGame NPCLLM
0 likes · 3 min read

This guide shows how to accelerate Bloom‑7B1 inference on Alibaba Cloud ACK by converting the model to FasterTransformer format, deploying it with Triton Server, and comparing performance against the original HuggingFace checkpoint, achieving roughly a 2.5‑fold speedup.
Bloom-7B1FasterTransformerKubernetes
0 likes · 17 min read

Tencent Cloud’s OCR team cut average response time from 1.8 seconds to under one second and boosted throughput by over 50 % by redesigning the model with self‑attention, accelerating inference with a Tensor‑Network accelerator, shrinking RPC payloads, enabling asynchronous logging, and optimizing multi‑region GPU memory utilization.
AI ModelOCRcloud services
0 likes · 13 min read

This talk presents practical methods for accelerating deep model inference, detailing two case studies—text QA and speech QA—along with their technical challenges, and outlines optimization strategies such as model compression, multi‑operator fusion, matrix multiplication tuning, quantization, and dynamic batching.
Operator Fusiondeep learningdynamic batching
0 likes · 12 min read


ITPUBApr 27, 2022 · Artificial Intelligence
This article details the design and optimization of 58.com’s WPAI machine learning platform, covering background, training‑task scheduling, elastic inference scaling, offline‑online resource mixing, and model‑inference acceleration, and shows how these techniques collectively raised GPU usage by 51% and CPU usage by 38% while cutting costs.
AI PlatformElastic ScalingGPU utilization
0 likes · 26 min read


58 TechJan 10, 2022 · Artificial Intelligence
This article details the 58.com WPAI machine learning platform's architecture and the optimizations applied to training task scheduling, inference service elastic scaling, and offline‑online resource mixing, demonstrating how these techniques significantly improve GPU/CPU utilization and inference performance across both GPU and CPU environments.
AIElastic ScalingKubernetes
0 likes · 27 min read

This article describes the design, automated workflow, functional modules, and performance results of a comprehensive AI model optimization platform built for Ctrip's travel business, covering operator libraries, graph optimization, model compression techniques such as distillation, quantization, pruning, and deployment integration.
AutoMLai-optimizationinference acceleration
0 likes · 16 min read


HomeTechSep 4, 2019 · Artificial Intelligence
This article explains how to use NVIDIA TensorRT to accelerate TensorFlow model inference by describing TensorRT architecture, optimization techniques such as layer fusion and precision calibration, detailing the conversion of frozen_graph and saved_model formats, presenting experimental setup and performance comparisons, and summarizing the achieved speed‑up.
TensorFlowTensorRTdeep learning
0 likes · 13 min read
