Tagged articles
LLM Deployment
35 articles · Page 1 of 1
Unsloth's Dynamic v3.1 quantization reduces Qwen3.8-Flash-Next to 68GB (1-bit) and DeepSeek-V4.1-Flash-1017 to 128-145GB (1-bit), enabling them to run on Macs with 96GB+ unified memory, while benchmarks show 50% fewer tokens, 7% faster speed, and 15% higher accuracy versus same-size quantizations.
DeepSeek-V4.1-Flash-1017Dynamic QuantizationLLM Deployment
0 likes · 11 min read

Kimi K3.1 Pro's 1.56TB MXFP4 weights are released but require 8×B300 GPUs minimum; a 256GB Mac Studio falls 6× short, no GGUF quantization exists yet, and inference on similar MoE architecture hits 0.1 tok/s due to memory bandwidth bottlenecks, making local deployment impractical for now.
Kimi K3.1 ProLLM DeploymentMXFP4
0 likes · 17 min read

Analysis of Unsloth's V3.1 quantization for Qwen3.8-Flash-Next reveals the 15% accuracy gain is versus other quantizations not the base model, 32GB Mac support relies on SSD offloading of a 51B n-gram table via mmap, and practical Agent workloads need 96GB+ unified memory.
GGUFLLM DeploymentMac unified memory
0 likes · 14 min read

This guide details five iron laws for deploying large language models on a 64GB MacBook Pro with M5 Pro chip, covering memory budgeting (45GB usable), model tier recommendations (32B smooth, 70B borderline), inference engine selection, speedup techniques like speculative decoding, and API serving best practices.
Apple SiliconLLM DeploymentMacBook Pro M5 Pro
0 likes · 10 min read

MiaAI-Lab demonstrates running the 552B-parameter DeepSeek-V4.1-Flash-1017 model on two NVIDIA DGX Spark workstations (256GB unified memory) using a Zig-based runtime, ConnectX-7 200 Gb/s interconnect, EXL3 2.9 bpw quantization, and local NVMe storage with memory guards, achieving 75 tok/s single-stream and 130 tok/s 4-stream decode throughput with 512K context.
ConnectX-7DGX SparkDeepSeek-V4.1-Flash-1017
0 likes · 14 min read

This comprehensive guide details a five-step methodology for assessing GPU, VRAM, storage, and network requirements when deploying large language models privately, covering weight and KV cache calculations, hardware selection trade-offs, and validation commands.
GPU capacity planningInference OptimizationKV Cache
0 likes · 89 min read


liandkOct 13, 2026 · Artificial Intelligence
This article details real-world pitfalls when deploying large language models locally on Mac using Ollama, covering memory overflow causes, key parameter tuning (num_ctx, num_parallel, num_gpu), quantization level selection, model recommendations for coding vs chat, and monitoring strategies to avoid swap-induced performance degradation.
LLM DeploymentMacOllama
0 likes · 14 min read

The author covers Alibaba Qwen team's upcoming Qwen4 family with open-source plans, surveys recent global AI model releases, notes a slowdown in Chinese model updates, shares hands-on testing of Qwen3.8-27B quantization variants on A100 and 4090 GPUs, and recommends FP8 for dual-4090 and GGUF UD-Q4+ for single-4090 setups.
FP8GGUFLLM Deployment
0 likes · 4 min read

This comprehensive tutorial covers multi-GPU inference for large language models, detailing VRAM estimation formulas, parallel strategies (tensor, pipeline, data), GPU interconnects (NVLink, PCIe), framework comparisons (vLLM, DeepSpeed, Transformers), quantization techniques (GPTQ, AWQ, bitsandbytes), performance benchmarking, and production deployment with Docker and Kubernetes.
DeepSpeedGPU memory optimizationLLM Deployment
0 likes · 32 min read

PrismML's Ternary Bonsai 2 27B uses rotated weight basis and FP16 group-wise scaling to ternary-quantize Qwen3.8-27B to 1.76 bits (5.9GB), achieving 98.2% benchmark retention across coding, reasoning, and agent tasks, with custom CUDA/MLX kernels enabling fast inference on consumer GPUs and Apple Silicon.
Bonsai 2GGUFLLM Deployment
0 likes · 15 min read

This guide details a 10-step framework for safely deploying changes to LLM model, prompt, and tokenizer components using canary releases, traffic routing, automated metrics comparison, and instant rollback mechanisms with Kubernetes, Istio, and Prometheus.
IstioKubernetesLLM Deployment
0 likes · 44 min read

This article explains how INT8, INT4, bitsandbytes, GPTQ, and AWQ quantization methods can dramatically cut memory usage, boost inference speed, and lower costs for large language models, while detailing their trade‑offs, practical workflows, benchmark results, and common pitfalls to help engineers decide which technique best fits their production scenario.
AWQGPTQINT4
0 likes · 22 min read

Small models are chosen based on deployment constraints rather than absolute parameter counts; the article outlines how resource limits, latency, cost, privacy, and task characteristics define their suitability, compares their strengths and weaknesses to large models, and offers system‑level design patterns for effective use.
Inference OptimizationLLM DeploymentRAG
0 likes · 20 min read

The article details how to deploy the 4‑bit quantized Qwen3.6-35B model with vLLM 0.17 (and 0.19.1 patch) on a Docker container, compares its memory usage and token‑generation speed to Qwen3.5‑35B, and shares practical scripts and observed performance of roughly 150 tokens per second.
DockerLLM DeploymentQwen3.6
0 likes · 5 min read

The article walks through upgrading vLLM to 0.17.0, configuring Docker containers for 4090 GPUs, comparing FP8 and 4‑bit quantization of Qwen3.5 35B and 27B models, and presents detailed performance numbers and script parameters that reveal trade‑offs in memory usage and throughput.
4-bit quantizationDockerFP8
0 likes · 7 min read

This article explains the challenges large language models face in real‑world applications, introduces Retrieval‑Augmented Generation (RAG) as a solution, and details a modular RAG architecture, its components, and practical techniques for document parsing, query rewriting, hybrid retrieval, ranking, and answer generation in an enterprise setting.
LLM DeploymentRAGRanking Models
0 likes · 22 min read

This article explains how to run the 397B Qwen3.5 model on a Mac by using Unsloth Dynamic 2.0 quantization (2‑bit, 3‑bit, or 4‑bit), outlines hardware requirements, provides compilation and download commands for llama.cpp, shows how to launch inference in thinking and non‑thinking modes, and compares several deployment options such as llama‑server, Transformers, SGLang/vLLM, and MLX.
Dynamic QuantizationGGUFLLM Deployment
0 likes · 14 min read

The article reviews the newly released quantized versions of the Kimi K2.5 large language model, detailing hardware needs, recommended quantization levels, deployment steps on Apple MLX and Inferencer, performance numbers, and the model's hybrid thinking mode.
InferencerKimi K2.5LLM Deployment
0 likes · 5 min read

This guide explains DeepSeek R1’s model variants, hardware requirements, local installation steps using Ollama, LM Studio or Docker, and how to add visual interfaces like Open‑WebUI and Dify for a complete on‑premise AI solution.
DeepSeekDifyLLM Deployment
0 likes · 14 min read

This guide explains how to prepare, deploy, and verify the Qwen3‑8B large language model on an Alibaba Cloud Container Service for Kubernetes (ACK) cluster using ACS GPU resources, covering prerequisites, model download, storage setup, Kubernetes manifests, and testing the inference service.
ACKACSGPU
0 likes · 8 min read

This article provides a comprehensive technical guide for privately deploying DeepSeek large language models, covering model and runtime parameter selection, hardware sizing calculations, software stack preparation, inference service setup, performance tuning, and security monitoring considerations.
AI hardware sizingDeepSeekInference Optimization
0 likes · 14 min read


Qborfy AIMar 27, 2025 · Artificial Intelligence
This article walks through the entire process of deploying the DeepSeek‑R1 large language model on a personal machine, covering hardware requirements, Ollama installation, model download, service startup, remote access configuration, and visual UI integration with Dify, complete with concrete commands and screenshots.
AIDeepSeekDocker
0 likes · 9 min read


AIWalkerFeb 27, 2025 · Artificial Intelligence
This article provides a comprehensive, hands‑on guide for installing and configuring DeepSeek‑R1 with Ollama and vLLM, setting up multi‑node multi‑GPU environments, running performance benchmarks, optimizing runtime parameters, and even generating a full PyTorch distributed‑training script.
DeepSeek-R1GPU optimizationLLM Deployment
0 likes · 39 min read

This article provides a comprehensive tutorial on locally deploying the 671‑billion‑parameter DeepSeek R1 model using Ollama, covering model selection, hardware requirements, dynamic quantization, detailed installation steps, performance observations, and practical recommendations for consumer‑grade hardware.
AI Model OptimizationDeepSeekDynamic Quantization
0 likes · 14 min read


21CTOFeb 16, 2025 · Artificial Intelligence
This guide walks you through setting up a local DeepSeek large language model, covering environment preparation, model acquisition, dependency installation, FastAPI service creation, Docker containerization, optional front‑end interface, performance tuning, and common troubleshooting steps.
AI ModelDeepSeekDocker
0 likes · 7 min read

This guide walks you through the reasons for deploying a private DeepSeek large‑model, compares full and distilled versions, shows how to purchase a JD Cloud computer, install Ollama, run the model, and integrate a local knowledge base using CherryStudio, Page Assist, and Anything LLM.
AI ModelDeepSeekJD Cloud
0 likes · 17 min read

This tutorial walks developers through installing Ollama on a Huawei cloud VM, deploying the DeepSeek large language model, integrating it with the ChatBox web UI, and finally connecting the model to CodeArts IDE via the Continue plugin for seamless AI‑assisted coding.
ChatboxCodeArts IDEDeepSeek
0 likes · 5 min read

This guide walks you through preparing a JD Cloud GPU instance, installing NVIDIA drivers, deploying Ollama, running the DeepSeek LLM (including model download and execution), configuring the Chatbox graphical client for interactive queries, and optionally feeding local documents into AnythingLLM for a private knowledge base.
ChatboxDeepSeekGPU cloud
0 likes · 17 min read

This guide explains how to deploy the full 671B DeepSeek R1 model on local hardware using Ollama, leveraging dynamic quantization to shrink model size, detailing hardware requirements, step‑by‑step installation, configuration, performance observations, and practical recommendations.
DeepSeekDynamic QuantizationGPU
0 likes · 12 min read

Facing frequent overloads on DeepSeek's official service, this guide details how to run DeepSeek locally with Ollama, deploy it on major cloud platforms such as Huawei, Alibaba, Tencent, Baidu and ZStack, integrate it into enterprise private clusters, leverage open‑source tools like HuggingFace, vLLM and Dify, and showcases real‑world applications in finance, education, and cross‑domain testing.
DeepSeekLLM DeploymentOllama
0 likes · 10 min read


JavaEdgeNov 20, 2024 · Artificial Intelligence
The article explains why deploying large language models is challenging and presents seven practical techniques—including defining deployment boundaries, model quantization, inference optimization, infrastructure consolidation, model replacement planning, GPU utilization, and using smaller models—to make LLM deployment more efficient and cost‑effective.
GPU optimizationLLM DeploymentModel Scaling
0 likes · 24 min read


JavaEdgeOct 14, 2024 · Artificial Intelligence
This article introduces LangServe, explains its key features for LLM deployment, walks through environment setup, shows how to build a FastAPI‑based REST API with code examples, demonstrates testing via Postman and remote client calls, and summarizes its benefits for AI model serving.
AI model servingFastAPILLM Deployment
0 likes · 9 min read

This guide walks you through deploying the ChatGLM3‑6B large language model locally, adding the M3E vector embedding model, setting up One‑API and FastGPT with Docker, configuring environments, fine‑tuning with LoRA, and testing the integrated knowledge‑base Q&A system.
ChatGLM3DockerFastGPT
0 likes · 15 min read


21CTOApr 23, 2024 · Artificial Intelligence
This guide explains how to deploy open‑source large language models using vLLM, benchmark latency and throughput, and apply 8‑bit/4‑bit quantization techniques such as BitsandBytes and NF4 to achieve faster inference on limited‑GPU hardware.
LLM DeploymentPythonlarge language models
0 likes · 13 min read

This presentation explains how OpenLLM, an open‑source LLM framework, together with BentoML, addresses the challenges of deploying large language models by offering model switching, memory optimizations, multi‑GPU support, observability, and easy containerized deployment for production AI applications.
BentoMLLLM DeploymentOpenLLM
0 likes · 18 min read
