BestHub
Sign in
Collection size
95 articles
Page 5 of 5
AI Engineer Programming
AI Engineer Programming
Aug 21, 2026 · Artificial Intelligence

Essential Concepts and Terminology for Deploying Large Language Models Locally

This article walks through the core concepts needed before deploying a large language model on‑premises, covering weight precision, quantization methods, model packaging formats, inference engines, GPU memory considerations, KV‑cache sizing, sampling strategies, optional extensions such as LoRA and RAG, and a step‑by‑step decision workflow to match hardware, model, and deployment goals.

DeploymentKV cacheLLM
0 likes · 21 min read
Essential Concepts and Terminology for Deploying Large Language Models Locally
ThinkingAgent
ThinkingAgent
Jun 22, 2026 · Artificial Intelligence

How to Achieve Full‑Stack AI Observability: Tracking Prompts, Tool Calls, Traces, and Tokens

The article explains why modern LLM‑based AI systems are opaque, defines AI observability as a four‑dimensional practice (Prompt, Tool Call, Trace, Token), and provides concrete architectures, code samples, best‑practice checklists, and real‑world case studies to turn black‑box AI into a transparent, monitorable service.

AI ObservabilityLangfusePrompt tracking
0 likes · 30 min read
How to Achieve Full‑Stack AI Observability: Tracking Prompts, Tool Calls, Traces, and Tokens
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Nov 19, 2025 · Artificial Intelligence

Boost LLM Inference Speed with Token‑Level Two‑Chunk Overlap

Token‑level Two‑Chunk Overlap replaces traditional batch‑level Two‑Batch Overlap, dynamically splitting sequences into balanced token chunks, enabling near‑equal compute and communication times, improving GPU utilization and achieving up to 30% throughput gains in heterogeneous request workloads, with zero accuracy loss.

GPU utilizationLLM inferenceSGLang
0 likes · 9 min read
Boost LLM Inference Speed with Token‑Level Two‑Chunk Overlap
DaTaobao Tech
DaTaobao Tech
Sep 27, 2023 · Artificial Intelligence

FlashAttention-2: Efficient Attention Algorithm for Transformer Acceleration and AIGC Applications

FlashAttention‑2 is an IO‑aware exact attention algorithm that cuts GPU HBM traffic through tiling and recomputation, optimizes non‑matmul FLOPs, expands sequence‑parallelism and warp‑level work distribution, delivering up to 2× speedup over FlashAttention, near‑GEMM efficiency, and enabling longer‑context Transformer training and inference for AIGC with fastunet and negligible accuracy loss.

AIGCAttention optimizationFlashAttention-2
0 likes · 20 min read
FlashAttention-2: Efficient Attention Algorithm for Transformer Acceleration and AIGC Applications
Old Zhang's AI Learning
Old Zhang's AI Learning
Mar 7, 2026 · Artificial Intelligence

vLLM 0.17.0 Release: Full Qwen 3.5 Support and Anthropic API Compatibility

The vLLM 0.17.0 release brings FlashAttention 4 integration, a mature Model Runner V2, complete Qwen 3.5 series support, a one‑click performance‑mode flag, Anthropic API compatibility, advanced weight‑offloading, broader hardware support beyond NVIDIA, ASR model integration, and detailed upgrade and installation guidance.

ASRAnthropic APIFlashAttention
0 likes · 12 min read
vLLM 0.17.0 Release: Full Qwen 3.5 Support and Anthropic API Compatibility
Geek Labs
Geek Labs
Jun 29, 2026 · Artificial Intelligence

DeepSpec Boosts Large-Model Inference Speed by 2–5× with Speculative Decoding

DeepSpec, an open‑source framework from DeepSeek, accelerates large‑language‑model inference by 2–5× through speculative decoding, where a lightweight draft model generates candidate tokens that the target model validates in parallel, reducing the serial bottleneck of autoregressive decoding and offering a full‑stack pipeline from data preparation to evaluation.

DeepSpecGPUPython
0 likes · 6 min read
DeepSpec Boosts Large-Model Inference Speed by 2–5× with Speculative Decoding
Old Zhang's AI Learning
Old Zhang's AI Learning
Jul 11, 2026 · Artificial Intelligence

Unsloth’s Dynamic NVFP4 Makes Qwen3.6 Run 2.5× Faster Than NVIDIA’s Official Quantization

Unsloth’s Dynamic NVFP4 quantization (W4A4) lets Qwen3.6‑27B run up to 2.5× faster on Blackwell GPUs while keeping near‑BF16 accuracy, adds FP8 KV‑Cache calibration, provides detailed hardware requirements, benchmark tables, and step‑by‑step deployment guides via vLLM, SGLang or Unsloth Studio.

Blackwell GPUDynamic QuantizationFP8 KV cache
0 likes · 13 min read
Unsloth’s Dynamic NVFP4 Makes Qwen3.6 Run 2.5× Faster Than NVIDIA’s Official Quantization
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
Jan 21, 2026 · Artificial Intelligence

Boost LLM Performance: Deploy Qwen3‑235B with PD‑Separation, MoE, SGLang & RBG

This article details how to deploy the 235‑billion‑parameter Qwen3‑235B model using PD‑separation and MoE techniques, explains the associated challenges, and demonstrates a production‑grade solution built on the high‑performance SGLang inference engine and the RoleBasedGroup (RBG) orchestration framework, complete with benchmark results and best‑practice YAML examples.

AIKubernetesLLM
0 likes · 21 min read
Boost LLM Performance: Deploy Qwen3‑235B with PD‑Separation, MoE, SGLang & RBG
Baobao Algorithm Notes
Baobao Algorithm Notes
Jan 7, 2025 · Artificial Intelligence

How Efficient Is DeepSeek V3? Calculating Its MFU Around 37%

This article derives DeepSeek V3's training Model FLOPs Utilization (MFU) using publicly available data, showing an MFU of roughly 37%—about a 60% improvement over V2—and provides detailed formulas, parameter settings, and a reproducible Python script.

AI performanceDeepSeekMFU
0 likes · 8 min read
How Efficient Is DeepSeek V3? Calculating Its MFU Around 37%
Old Zhang's AI Learning
Old Zhang's AI Learning
Oct 20, 2026 · Artificial Intelligence

Deploying 552B DeepSeek-V4.1-Flash on Dual DGX Sparks: 130 tok/s, 2.9 bpw Quantization

MiaAI-Lab demonstrates running the 552B-parameter DeepSeek-V4.1-Flash-1017 model on two NVIDIA DGX Spark workstations (256GB unified memory) using a Zig-based runtime, ConnectX-7 200 Gb/s interconnect, EXL3 2.9 bpw quantization, and local NVMe storage with memory guards, achieving 75 tok/s single-stream and 130 tok/s 4-stream decode throughput with 512K context.

ConnectX-7DGX SparkDeepSeek-V4.1-Flash-1017
0 likes · 14 min read
Deploying 552B DeepSeek-V4.1-Flash on Dual DGX Sparks: 130 tok/s, 2.9 bpw Quantization
Old Zhang's AI Learning
Old Zhang's AI Learning
May 13, 2026 · Artificial Intelligence

Why vLLM Now Leads Open‑Source LLM Inference Benchmarks

vLLM tops the Artificial Analysis ranking by delivering the highest throughput for DeepSeek V3.2, Qwen 3.5 397B, and MiniMax‑M2.5 on identical NVIDIA Blackwell Ultra hardware, thanks to extensive kernel‑fusion optimizations that remain in the main branch.

DeepSeekLLM inferenceQwen
0 likes · 7 min read
Why vLLM Now Leads Open‑Source LLM Inference Benchmarks
Alimama Tech
Alimama Tech
Feb 12, 2025 · Artificial Intelligence

HighService: A High‑Performance Pythonic AI Service Framework for Model Inference and Global Resource Scheduling

HighService, Alibaba’s Pythonic AI service framework, accelerates large‑model inference and maximizes GPU utilization by separating CPU‑GPU processes, offering out‑of‑the‑box quantization, parallelism and caching, and dynamically reallocating idle GPUs across clusters through a master‑worker scheduler to keep online latency low while boosting offline throughput for diffusion and LLM workloads.

AI ServiceHigh PerformancePython
0 likes · 16 min read
HighService: A High‑Performance Pythonic AI Service Framework for Model Inference and Global Resource Scheduling