Collection size

95 articles

Page 5 of 5

May 25, 2026 · Artificial Intelligence

vLLM Deep Dive: Continuous Batching and Paged Attention for Fast LLM Inference

This article walks through a two‑month source‑code study of vLLM, explaining how token‑level scheduling, continuous batching, and the Paged Attention mechanism reshape tensor dimensions to turn large‑model inference into a compute‑bound, high‑throughput process while managing GPU memory efficiently.

Continuous BatchingFlashAttentionGPU Optimization

0 likes · 29 min read

vLLM Deep Dive: Continuous Batching and Paged Attention for Fast LLM Inference

Geek Labs

Mar 24, 2026 · Industry Insights

9 Must‑See GitHub Projects: MacBook‑Run LLM, WeChat AI, Multi‑Agent Collaboration and More

This article reviews nine standout GitHub open‑source projects, covering a C/Metal LLM engine for MacBooks, a Claude Code commercial‑analysis skill, multi‑agent communication tools, web‑enabled AI, autonomous research automation, WeChat AI integration, a minimalist terminal, a Codex console, and a lightweight WARP proxy.

AIDockerGitHub

0 likes · 10 min read

9 Must‑See GitHub Projects: MacBook‑Run LLM, WeChat AI, Multi‑Agent Collaboration and More

Machine Heart

Jul 3, 2026 · Artificial Intelligence

Avoiding Pitfalls in Heterogeneous Token Factories: Industry‑Level Design Practices for Cross‑Hardware LLM Inference

The article analyzes a recent multi‑institution paper that maps the design space of heterogeneous Prefill‑Decode LLM inference, identifies three core boundary decisions, presents nine deployment best practices, and validates them with a production token‑factory case on MuXi C600 and NVIDIA Hopper GPUs.

KV cacheLLMdeployment best practices

0 likes · 11 min read

Avoiding Pitfalls in Heterogeneous Token Factories: Industry‑Level Design Practices for Cross‑Hardware LLM Inference

Alibaba Cloud Infrastructure

Feb 13, 2025 · Artificial Intelligence

Deploying DeepSeek‑R1 671B Distributed Inference Service on Alibaba Cloud ACK with vLLM and Dify

This article explains how to quickly deploy the full‑parameter DeepSeek‑R1 671B model in a multi‑node GPU‑enabled Kubernetes cluster on Alibaba Cloud ACK, covering prerequisites, model parallelism, vLLM‑Ray distributed deployment, service verification, and integration with Dify to build a private AI Q&A assistant.

DeepSeekDifyKubernetes

0 likes · 12 min read

Deploying DeepSeek‑R1 671B Distributed Inference Service on Alibaba Cloud ACK with vLLM and Dify

Alibaba Cloud Native

Jul 30, 2024 · Cloud Native

Deploy ComfyUI as a Serverless API for Scalable AI Image Generation

This article explains how to transform ComfyUI into a serverless API using Alibaba Cloud Function Compute, detailing the challenges of GPU resource costs, high concurrency, and usability, while providing a step‑by‑step guide, code examples, and best‑practice recommendations for building scalable AI drawing applications.

AI image generationAPIComfyUI

0 likes · 21 min read

Deploy ComfyUI as a Serverless API for Scalable AI Image Generation

Geek Labs

May 13, 2026 · Artificial Intelligence

Two LLM Inference Acceleration Projects: A Mac‑Local Engine vs a Data‑Center Engine

This article compares two recent GitHub LLM inference engines—ds4.c, a Metal‑optimized engine for DeepSeek V4 Flash on Apple Silicon Macs, and TokenSpeed, a Python/C++‑based, data‑center‑grade engine for GPU clusters—detailing their design choices, performance numbers, usage instructions, and suitable scenarios.

DeepSeekGPULLM

0 likes · 8 min read

Two LLM Inference Acceleration Projects: A Mac‑Local Engine vs a Data‑Center Engine

Old Zhang's AI Learning

Apr 26, 2026 · Artificial Intelligence

Why Deploying DeepSeek‑V4 Locally with vLLM Is So Challenging

The article dissects DeepSeek‑V4’s local deployment using vLLM, explaining the steep hardware requirements, the complex heterogeneous KV‑cache architecture, and the aggressive kernel‑fusion and multi‑stream optimizations that together make high‑context inference both memory‑intensive and engineering‑heavy.

DeepSeek V4GPU memoryKV cache

0 likes · 15 min read

Why Deploying DeepSeek‑V4 Locally with vLLM Is So Challenging

Alibaba Cloud Developer

Jan 15, 2026 · Artificial Intelligence

How Hierarchical Sparse Attention Breaks KVCache Limits for Ultra‑Long Context LLMs

This article explains how a hierarchical sparse‑attention framework redesigns KVCache storage across GPU, CPU, and remote memory, eliminates bandwidth and capacity bottlenecks, and enables efficient inference for 128K‑token and larger contexts with dramatically reduced GPU memory usage and higher throughput.

Dynamic Sparse AttentionGPU memory optimizationHierarchical Storage

0 likes · 20 min read

How Hierarchical Sparse Attention Breaks KVCache Limits for Ultra‑Long Context LLMs

DataFunSummit

Mar 14, 2025 · Artificial Intelligence

Insights from Zhihu's ZhiLight Large‑Model Inference Framework: Architecture, Parallelism, and Performance Optimizations

The article summarizes Zhihu's machine‑learning platform lead Wang Xin's presentation on the ZhiLight large‑model inference framework, covering model execution mechanisms, GPU workload analysis, pipeline and tensor parallelism, GPU architecture evolution, open‑source engine comparisons, ZhiLight's compute‑communication overlap and quantization optimizations, benchmark results, supported models, and future directions.

GPULLMOpen‑source

0 likes · 13 min read

Insights from Zhihu's ZhiLight Large‑Model Inference Framework: Architecture, Parallelism, and Performance Optimizations

DataFunSummit

Dec 14, 2020 · Artificial Intelligence

LightSeq: High‑Performance Open‑Source Inference Engine for Transformers, GPT and Other NLP Models

This article introduces LightSeq, an open‑source, GPU‑accelerated inference engine that dramatically speeds up Transformer‑based models such as BERT and GPT by up to 14× over TensorFlow, supports multiple decoding strategies, integrates seamlessly with major deep‑learning frameworks, and provides detailed performance benchmarks and technical optimizations.

Deep LearningGPULightSeq

0 likes · 15 min read

LightSeq: High‑Performance Open‑Source Inference Engine for Transformers, GPT and Other NLP Models

Alibaba Cloud Native

Jan 17, 2024 · Artificial Intelligence

Boost LLM Inference with TensorRT‑LLM on Alibaba Cloud ACK: A Step‑by‑Step Guide

This article explains how TensorRT‑LLM accelerates large language model inference by applying quantization, in‑flight batching, advanced attention variants, and graph rewriting, and walks through a complete deployment on Alibaba Cloud Container Service (ACK) with environment setup, model compilation, benchmarking, and performance comparison.

Cloud Native AIIn‑Flight BatchingQuantization

0 likes · 13 min read

Boost LLM Inference with TensorRT‑LLM on Alibaba Cloud ACK: A Step‑by‑Step Guide

Alibaba Cloud Infrastructure

Mar 17, 2025 · Cloud Native

Boost LLM Inference with ACK Gateway AI Extension: A Step‑by‑Step Guide

This guide demonstrates how to deploy the QwQ‑32B large language model on an Alibaba Cloud ACK cluster, configure OSS storage, enable the ACK Gateway with AI Extension, set up InferencePool and InferenceModel resources, and benchmark intelligent routing versus standard gateway routing, revealing latency and throughput improvements.

ACK GatewayAI ExtensionKubernetes

0 likes · 16 min read

Boost LLM Inference with ACK Gateway AI Extension: A Step‑by‑Step Guide

Alibaba Cloud Infrastructure

Feb 13, 2025 · Cloud Computing

Deploy DeepSeek‑R1 LLM on Alibaba Cloud ACK One with ACS GPU in Minutes

This guide walks you through deploying the DeepSeek‑R1 large‑language‑model inference service on Alibaba Cloud ACK One registered clusters using ACS GPU compute, covering model preparation, OSS storage setup, PersistentVolume configuration, arena‑based service deployment, and verification steps with concrete commands and parameters.

ACK OneACS GPUDeepSeek

0 likes · 14 min read

Deploy DeepSeek‑R1 LLM on Alibaba Cloud ACK One with ACS GPU in Minutes

James' Growth Diary

May 14, 2026 · Artificial Intelligence

LLM Semantic Routing Explained: Model‑Based Intent Classification and Three Keyword‑Matching Pitfalls

This article breaks down LLM semantic routing as a classifier, compares keyword, embedding, and LLM‑based routes, provides full TypeScript implementations, introduces hybrid routing for speed and accuracy, and covers production‑grade observability and dynamic configuration to avoid common pitfalls.

Hybrid RoutingLLMLangChain

0 likes · 33 min read

LLM Semantic Routing Explained: Model‑Based Intent Classification and Three Keyword‑Matching Pitfalls

ITPUB

Jul 25, 2026 · Artificial Intelligence

Running a 35B MoE Model at 32 tokens/s on $600 Tesla P100 GPUs

This article details how to repurpose two second‑hand Tesla P100 GPUs (≈$600 each) in a Dell R730XD server with ESXi 8.0, AlmaLinux 10, NVIDIA drivers, Docker and Ollama to run the qwen3.6:35b model at 32.77 tokens/s, including step‑by‑step configuration, performance benchmarks, and practical tips.

DockerESXiGPU passthrough

0 likes · 14 min read

Running a 35B MoE Model at 32 tokens/s on $600 Tesla P100 GPUs