Collection size

95 articles

Page 4 of 5

Mar 27, 2025 · Cloud Native

Deploy the QwQ‑32B LLM on Alibaba Cloud Function Compute with CAP in Minutes

This guide walks you through deploying the open‑source QwQ‑32B model on Alibaba Cloud Function Compute using the Cloud Application Platform (CAP), covering architecture, required services, account setup, step‑by‑step deployment, cost considerations, model interaction via Open WebUI and Chatbox, scaling configuration, and resource cleanup.

CAPFunction ComputeOpen WebUI

0 likes · 8 min read

Deploy the QwQ‑32B LLM on Alibaba Cloud Function Compute with CAP in Minutes

Old Zhang's AI Learning

Jul 12, 2026 · Artificial Intelligence

vLLM 0.25 Release: New Model Support, Multi‑Hardware, and Major Performance Boosts

The vLLM 0.25 update adds dozens of new model architectures, expands support to AMD ROCm, CPUs, Intel XPU and IBM Power, fixes speculative decoding for reasoning models, and introduces a suite of performance, KV‑cache, quantization, and API enhancements for large‑scale LLM inference.

AMD ROCmGPU optimizationKV cache

0 likes · 8 min read

vLLM 0.25 Release: New Model Support, Multi‑Hardware, and Major Performance Boosts

Old Zhang's AI Learning

Jul 2, 2026 · Artificial Intelligence

vLLM 0.24.0 Release: New Features for Faster, Memory‑Efficient Large‑Model Deployment

The vLLM 0.24.0 update adds MiniMax‑M3, DeepSeek‑V4, DiffusionGemma support, a Streaming Parser Engine, and a new device_ids parameter, delivering faster inference, lower memory use, and broader hardware compatibility for large‑model deployments.

DeepSeek-V4DiffusionGemmaLarge Language Models

0 likes · 9 min read

vLLM 0.24.0 Release: New Features for Faster, Memory‑Efficient Large‑Model Deployment

Old Zhang's AI Learning

May 1, 2026 · Artificial Intelligence

NVIDIA’s Open‑Source Multimodal Nemotron 3 Nano Omni: Run Locally on Consumer GPUs (English‑Only)

NVIDIA’s Nemotron 3 Nano Omni 30B‑A3B‑Reasoning model, an open‑source multimodal LLM with 30 B parameters, 256K context and video‑audio‑image‑text capabilities, outperforms comparable models by up to 9.2× in video throughput, runs on consumer GPUs via 4‑bit GGUF quantization, but currently supports only English input.

GGUFGPUNVIDIA

0 likes · 17 min read

NVIDIA’s Open‑Source Multimodal Nemotron 3 Nano Omni: Run Locally on Consumer GPUs (English‑Only)

ByteDance Cloud Native

Mar 7, 2025 · Artificial Intelligence

How to Deploy the QwQ-32B Large Language Model on Volcengine Cloud in Minutes

This guide walks you through the end‑to‑end process of deploying the open‑source QwQ‑32B inference model on Volcengine's cloud platform, covering GPU ECS selection, VKE cluster creation, continuous delivery CP setup, vLLM service launch, and API gateway exposure.

GPU ECSQwQ-32BVKE

0 likes · 8 min read

How to Deploy the QwQ-32B Large Language Model on Volcengine Cloud in Minutes

Old Zhang's AI Learning

Apr 20, 2026 · Artificial Intelligence

Qwen3.6-35B Quantized Model on vLLM: Local Deployment and Performance Benchmark

The article details how to deploy the 4‑bit quantized Qwen3.6-35B model with vLLM 0.17 (and 0.19.1 patch) on a Docker container, compares its memory usage and token‑generation speed to Qwen3.5‑35B, and shares practical scripts and observed performance of roughly 150 tokens per second.

DockerLLM deploymentQwen3.6

0 likes · 5 min read

Qwen3.6-35B Quantized Model on vLLM: Local Deployment and Performance Benchmark

AI Engineering

Aug 23, 2026 · Artificial Intelligence

FreeToken Runs a 35B MoE Model on an 8 GB GPU at 39 tokens/s – Is Local Freedom Real?

FreeToken, an open‑source MoE inference engine from UC Berkeley, enables a 35B model to run on an 8 GB GPU at 39.3 tokens per second by offloading weights to system memory and dynamically scheduling between CPU and GPU, offering a detailed performance analysis and community feedback.

CPU OffloadFreeTokenGPU

0 likes · 6 min read

FreeToken Runs a 35B MoE Model on an 8 GB GPU at 39 tokens/s – Is Local Freedom Real?

Old Zhang's AI Learning

Apr 25, 2026 · Artificial Intelligence

Deploying DeepSeek‑V4‑Flash Locally on 2 × NVIDIA H20 (96 GB) – Quick Performance Test

This article walks through deploying DeepSeek‑V4‑Flash on a server with two NVIDIA H20 GPUs (96 GB each), detailing model download, Docker image preparation, launch script tweaks, memory compression via FP8 and expert parallelism, and reports observed concurrency limits and token‑per‑second speeds, including a test that disables the model's thinking mode.

DeepSeek-V4DockerFP8 quantization

0 likes · 6 min read

Deploying DeepSeek‑V4‑Flash Locally on 2 × NVIDIA H20 (96 GB) – Quick Performance Test

MaGe Linux Operations

Jul 21, 2025 · Artificial Intelligence

Master Multi‑GPU Load Balancing for OLLAMA: From Zero to Production

This guide walks you through configuring OLLAMA for multi‑GPU load balancing, covering hardware checks, CUDA setup, native and Docker deployment methods, detailed parameter tuning, advanced sharding strategies, troubleshooting, performance optimization, and production‑grade monitoring to maximize throughput and stability of large language models.

AI deploymentCUDAPerformance Tuning

0 likes · 16 min read

Master Multi‑GPU Load Balancing for OLLAMA: From Zero to Production

Old Zhang's AI Learning

Oct 25, 2026 · Artificial Intelligence

vLLM & SGLang Surpass DeepSeek's DeepInfer: General Engines Prevail

vLLM and SGLang matched or exceeded DeepSeek's custom DeepInfer engine on DeepSeek-V4.1-Flash within days by adopting its open-sourced operators and optimizing for model-specific traits, proving general-purpose inference engines remain dominant due to full production-grade systems.

DeepInferDeepSeekKV cache

0 likes · 15 min read

vLLM & SGLang Surpass DeepSeek's DeepInfer: General Engines Prevail

Ubuntu

Sep 25, 2026 · Artificial Intelligence

Ollama vs llama.cpp on Ubuntu: Choose Convenience or Performance? Plus the Context Window Trap

This guide compares Ollama and llama.cpp for running local LLMs on Ubuntu, covering installation, the critical context window default pitfall, hardware requirements, model selection tips, and when to switch from Ollama's convenience to llama.cpp's performance tuning.

Context WindowGGUFGPU offloading

0 likes · 11 min read

Ollama vs llama.cpp on Ubuntu: Choose Convenience or Performance? Plus the Context Window Trap

Machine Learning Algorithms & Natural Language Processing

Oct 11, 2026 · Artificial Intelligence

DeepSeek's Cost Curve Chronicle: How Token Economics Rewrote LLM Architecture

This article chronicles DeepSeek's paper timeline from 2023 to 2026, analyzing how each model generation optimized training compute, KV cache, and throughput via MoE, MLA, sparse attention, and conditional memory, shifting long-context competition from maximum length to per-token byte cost.

DeepSeekGRPOKV-Cache Optimization

0 likes · 37 min read

DeepSeek's Cost Curve Chronicle: How Token Economics Rewrote LLM Architecture

Alibaba Cloud Native

Dec 19, 2024 · Artificial Intelligence

Deploy Open-Source LLMs on Alibaba Cloud Function Compute in 10 Minutes

This guide explains how to quickly launch an open‑source large language model from ModelScope on Alibaba Cloud Function Compute, covering the required cloud services, step‑by‑step deployment, reserved‑instance configuration, and how to invoke the model via the provided domain.

AIAlibaba CloudDeployment

0 likes · 7 min read

Deploy Open-Source LLMs on Alibaba Cloud Function Compute in 10 Minutes

Qborfy AI

Jun 15, 2026 · Artificial Intelligence

LLM API Parameter Comparison Across OpenAI, Claude, Gemini, DeepSeek, Kimi, MiniMax, Yi

This article provides a detailed side‑by‑side comparison of core API parameters such as temperature, top_p, top_k, penalties, max_tokens, tools and response_format across OpenAI, Claude, Gemini, DeepSeek, Kimi, MiniMax and Yi, explains common migration pitfalls, and offers practical guidance for selecting and adapting LLM services.

APILLMcompatibility

0 likes · 24 min read

LLM API Parameter Comparison Across OpenAI, Claude, Gemini, DeepSeek, Kimi, MiniMax, Yi

Old Zhang's AI Learning

Oct 7, 2026 · Artificial Intelligence

AI-Generated Inference Engine Beats vLLM by 90% at 1,792 tokens/s

Baseten engineers used Claude Code to auto-generate a custom inference engine (VibeQwen) for Qwen-3.6-35B-A3B on B200, achieving 1,792 tokens/s decode (90% faster than vLLM), 12ms TTFT, and 10,307 tokens/s throughput at concurrency 32, via MetaInfer's LLM-as-Compiler loop with accuracy oracle and reusable optimizations, also speeding up SAM 3.1 by 50% in two days.

AI-generated codeB200 GPUClaude Code

0 likes · 11 min read

AI-Generated Inference Engine Beats vLLM by 90% at 1,792 tokens/s

Alibaba Cloud Native

Dec 27, 2023 · Cloud Computing

One‑Click Deployment of LLMs to Alibaba Cloud Function Compute with SwingDeploy

This guide explains how to quickly select a ModelScope open‑source LLM, deploy it to Alibaba Cloud Function Compute using the SwingDeploy one‑click feature, enable reserved idle billing, and evaluate the cost savings compared with traditional GPU provisioning.

Function ComputeGPULLM

0 likes · 11 min read

One‑Click Deployment of LLMs to Alibaba Cloud Function Compute with SwingDeploy

Alibaba Cloud Big Data AI Platform

Jul 11, 2024 · Artificial Intelligence

How Llumnix Cuts LLM Serving Latency by 10× with Dynamic Scheduling

Alibaba Cloud's PAI team unveiled Llumnix, a dynamic scheduling framework for large language model serving that dramatically reduces tail latency, speeds high‑priority requests, and cuts costs, earning acceptance at OSDI 2024 and now open‑sourced on GitHub.

AI SystemsLLM servingLlumnix

0 likes · 5 min read

How Llumnix Cuts LLM Serving Latency by 10× with Dynamic Scheduling

Architect's Alchemy Furnace

Jul 17, 2025 · Artificial Intelligence

Explore the Ultimate Open-Source LLM Catalog: Models, Tools, and Resources

This article compiles a comprehensive, up‑to‑date inventory of open‑source large language models from Chinese and international organizations, detailing each model’s architecture, parameter count, multilingual capabilities, deployment requirements, and associated tools, offering a valuable reference for AI researchers and developers.

AILLMTools

0 likes · 50 min read

Explore the Ultimate Open-Source LLM Catalog: Models, Tools, and Resources

Black & White Path

Aug 2, 2026 · Artificial Intelligence

Running a 2.8‑Trillion‑Parameter K3 Model on 4 GB VRAM with AirLLM

AirLLM introduces layer‑wise inference and per‑expert streaming to decouple VRAM usage from model size, enabling the 2.8‑trillion‑parameter Kimi K3 LLM to run on a single consumer‑grade GPU while preserving full‑precision accuracy and offering security‑focused insights.

AirLLMKimi K3Large Language Models

0 likes · 9 min read

Running a 2.8‑Trillion‑Parameter K3 Model on 4 GB VRAM with AirLLM

AI Explorer

Mar 3, 2026 · Artificial Intelligence

How LMCache’s Lightning‑Fast KV Cache Slashes LLM First‑Token Latency

LMCache separates the KV cache from a vLLM instance into a shared service, dramatically cutting first‑token latency for repeated text, enabling multiple GPU instances to reuse cached vectors, improving hardware utilization, and supporting use cases such as long‑document QA, multi‑GPU load balancing, and prompt‑engineering, with a quick Docker‑based demo.

DockerKV cacheLLM inference

0 likes · 6 min read

How LMCache’s Lightning‑Fast KV Cache Slashes LLM First‑Token Latency