Tagged articles
TensorRT-LLM
16 articles · Page 1 of 1
This comprehensive guide explains inference frameworks for large language models, covering core concepts like Prefill/Decode and KV Cache, comparing major open-source frameworks (vLLM, SGLang, TensorRT-LLM), detailing request processing flow from scheduling to sampling, and identifying who benefits from understanding inference infrastructure.
KV CacheLLM servingMoE
0 likes · 46 min read

This article compares six popular LLM inference frameworks—vLLM, TensorRT‑LLM, llama.cpp, ds4.c, Ollama, and Omlx—across performance, ease of use, and hardware compatibility, then provides a practical matrix to help users select the best fit for their GPU.
Apple SiliconGPU performanceLLM inference
0 likes · 10 min read

TokenSpeed, an open‑source LLM inference engine designed for agent workloads, delivers TensorRT‑LLM‑level performance and vLLM‑level ease of use, outperforms TensorRT‑LLM by up to 11% throughput and halves latency on speculative decoding, and has earned Nvidia’s public recommendation.
LLM inferenceNVIDIA BlackwellTensorRT-LLM
0 likes · 8 min read

This article explains the NVFP4 4‑bit floating‑point quantization technique, shows how to deploy Qwen3‑30B‑A3B models with TensorRT‑LLM and vLLM, compares performance across NVFP4, AWQ and INT8 quantizations, and provides practical profiling commands for NVIDIA DGX systems.
LLMNVFP4NVIDIA DGX
0 likes · 23 min read

JD Advertising accelerates its generative‑recall recommendation system by integrating NVIDIA TensorRT‑LLM, which simplifies the pipeline, injects LLM knowledge, scales to billions of parameters, and delivers over five‑fold throughput gains, one‑fifth the cost, and significant CTR improvements in both recommendation and search.
Inference OptimizationLLMTensorRT-LLM
0 likes · 13 min read

This article explains how generative recommendation systems powered by large language models simplify the recommendation pipeline, integrate world knowledge, benefit from scaling laws, and require specialized engineering optimizations such as TensorRT‑LLM deployment, inference acceleration, and hybrid model strategies to achieve low latency and high throughput in real‑world e‑commerce scenarios.
AIInference OptimizationLLM
0 likes · 10 min read

This article explains how large language models reshape recommendation systems by simplifying pipelines, integrating world knowledge, and leveraging scaling laws, and details the engineering steps for deploying generative recall models—including product encoding, user prompting, model training, TensorRT‑LLM optimization, and continuous performance improvements.
LLMTensorRT-LLMai-optimization
0 likes · 13 min read

This article explains NVIDIA’s end‑to‑end stack for large language models, covering the NeMo Framework for data processing, training, and deployment, the open‑source TensorRT‑LLM inference accelerator, and the Retrieval‑Augmented Generation (RAG) technique that enriches model outputs with external knowledge.
NVIDIANeMoRAG
0 likes · 17 min read

This article introduces NVIDIA’s comprehensive solutions for large language models, covering the NeMo Framework’s full‑stack development pipeline, the open‑source TensorRT‑LLM inference accelerator, and Retrieval‑Augmented Generation techniques, while detailing data preprocessing, distributed training, model fine‑tuning, deployment, and performance optimizations.
NVIDIANeMo FrameworkRetrieval-Augmented Generation
0 likes · 16 min read

This guide walks through enabling KServe on Alibaba Cloud ASM, preparing the Llama‑2‑7B model with TensorRT‑LLM, creating the necessary Kubernetes resources, and deploying a serverless AI inference service that can be queried via a simple curl request.
AI inferenceKServeKubernetes
0 likes · 14 min read

This tutorial walks through deploying the Llama‑2‑7b‑hf model on Alibaba Cloud Kubernetes (ACK) using KServe, Triton Inference Server with the TensorRT‑LLM backend, covering prerequisites, model preparation, YAML configuration, PV/PVC setup, runtime creation, and troubleshooting steps.
AI inferenceKServeKubernetes
0 likes · 13 min read

This article presents a comprehensive overview of NVIDIA’s TensorRT-LLM, detailing its product positioning as a scalable LLM inference solution, key features such as model support, low-precision and quantization techniques, parallelism strategies, the end-to-end usage workflow, performance highlights, future roadmap, and answers to common technical questions.
LLM inferenceNVIDIAParallelism
0 likes · 13 min read

NVIDIA’s comprehensive LLM ecosystem combines the full‑stack NeMo Framework for data curation, distributed training, fine‑tuning, inference acceleration with TensorRT‑LLM and Triton, plus Retrieval‑Augmented Generation and Guardrails, enabling efficient, low‑latency, knowledge‑grounded model deployment across clusters.
AI accelerationModel TrainingNVIDIA
0 likes · 16 min read

This article explains NVIDIA’s end‑to‑end ecosystem for large language models, covering the NeMo Framework’s data processing, distributed training, model fine‑tuning, inference acceleration with TensorRT‑LLM, deployment via Triton, and Retrieval‑Augmented Generation (RAG) techniques that enhance model reliability and performance.
AINVIDIANeMo
0 likes · 16 min read

TensorRT-LLM is NVIDIA's scalable inference framework for large language models that combines TensorRT compilation, fast kernels, multi‑GPU parallelism, low‑precision quantization, and a PyTorch‑like API to deliver high‑performance LLM serving with extensive customization and future‑focused enhancements.
GPU accelerationLLM inferenceNVIDIA
0 likes · 12 min read

This article explains how TensorRT‑LLM accelerates large language model inference by applying quantization, in‑flight batching, advanced attention variants, and graph rewriting, and walks through a complete deployment on Alibaba Cloud Container Service (ACK) with environment setup, model compilation, benchmarking, and performance comparison.
Cloud-Native AIIn‑Flight BatchingLLM inference
0 likes · 13 min read
