The article details how to deploy the 4‑bit quantized Qwen3.6-35B model with vLLM 0.17 (and 0.19.1 patch) on a Docker container, compares its memory usage and token‑generation speed to Qwen3.5‑35B, and shares practical scripts and observed performance of roughly 150 tokens per second.
The article walks through upgrading vLLM to 0.17.0, configuring Docker containers for 4090 GPUs, comparing FP8 and 4‑bit quantization of Qwen3.5 35B and 27B models, and presents detailed performance numbers and script parameters that reveal trade‑offs in memory usage and throughput.
This article explains the challenges large language models face in real‑world applications, introduces Retrieval‑Augmented Generation (RAG) as a solution, and details a modular RAG architecture, its components, and practical techniques for document parsing, query rewriting, hybrid retrieval, ranking, and answer generation in an enterprise setting.
This article explains how to run the 397B Qwen3.5 model on a Mac by using Unsloth Dynamic 2.0 quantization (2‑bit, 3‑bit, or 4‑bit), outlines hardware requirements, provides compilation and download commands for llama.cpp, shows how to launch inference in thinking and non‑thinking modes, and compares several deployment options such as llama‑server, Transformers, SGLang/vLLM, and MLX.
The article reviews the newly released quantized versions of the Kimi K2.5 large language model, detailing hardware needs, recommended quantization levels, deployment steps on Apple MLX and Inferencer, performance numbers, and the model's hybrid thinking mode.
This guide explains DeepSeek R1’s model variants, hardware requirements, local installation steps using Ollama, LM Studio or Docker, and how to add visual interfaces like Open‑WebUI and Dify for a complete on‑premise AI solution.
This guide explains how to prepare, deploy, and verify the Qwen3‑8B large language model on an Alibaba Cloud Container Service for Kubernetes (ACK) cluster using ACS GPU resources, covering prerequisites, model download, storage setup, Kubernetes manifests, and testing the inference service.
This article provides a comprehensive technical guide for privately deploying DeepSeek large language models, covering model and runtime parameter selection, hardware sizing calculations, software stack preparation, inference service setup, performance tuning, and security monitoring considerations.
This article walks through the entire process of deploying the DeepSeek‑R1 large language model on a personal machine, covering hardware requirements, Ollama installation, model download, service startup, remote access configuration, and visual UI integration with Dify, complete with concrete commands and screenshots.
This article provides a comprehensive, hands‑on guide for installing and configuring DeepSeek‑R1 with Ollama and vLLM, setting up multi‑node multi‑GPU environments, running performance benchmarks, optimizing runtime parameters, and even generating a full PyTorch distributed‑training script.
This article provides a comprehensive tutorial on locally deploying the 671‑billion‑parameter DeepSeek R1 model using Ollama, covering model selection, hardware requirements, dynamic quantization, detailed installation steps, performance observations, and practical recommendations for consumer‑grade hardware.
This guide walks you through setting up a local DeepSeek large language model, covering environment preparation, model acquisition, dependency installation, FastAPI service creation, Docker containerization, optional front‑end interface, performance tuning, and common troubleshooting steps.
This guide walks you through the reasons for deploying a private DeepSeek large‑model, compares full and distilled versions, shows how to purchase a JD Cloud computer, install Ollama, run the model, and integrate a local knowledge base using CherryStudio, Page Assist, and Anything LLM.
This tutorial walks developers through installing Ollama on a Huawei cloud VM, deploying the DeepSeek large language model, integrating it with the ChatBox web UI, and finally connecting the model to CodeArts IDE via the Continue plugin for seamless AI‑assisted coding.
This guide walks you through preparing a JD Cloud GPU instance, installing NVIDIA drivers, deploying Ollama, running the DeepSeek LLM (including model download and execution), configuring the Chatbox graphical client for interactive queries, and optionally feeding local documents into AnythingLLM for a private knowledge base.
This guide explains how to deploy the full 671B DeepSeek R1 model on local hardware using Ollama, leveraging dynamic quantization to shrink model size, detailing hardware requirements, step‑by‑step installation, configuration, performance observations, and practical recommendations.
The article explains why deploying large language models is challenging and presents seven practical techniques—including defining deployment boundaries, model quantization, inference optimization, infrastructure consolidation, model replacement planning, GPU utilization, and using smaller models—to make LLM deployment more efficient and cost‑effective.
This article introduces LangServe, explains its key features for LLM deployment, walks through environment setup, shows how to build a FastAPI‑based REST API with code examples, demonstrates testing via Postman and remote client calls, and summarizes its benefits for AI model serving.
This guide walks you through deploying the ChatGLM3‑6B large language model locally, adding the M3E vector embedding model, setting up One‑API and FastGPT with Docker, configuring environments, fine‑tuning with LoRA, and testing the integrated knowledge‑base Q&A system.
This guide explains how to deploy open‑source large language models using vLLM, benchmark latency and throughput, and apply 8‑bit/4‑bit quantization techniques such as BitsandBytes and NF4 to achieve faster inference on limited‑GPU hardware.
This presentation explains how OpenLLM, an open‑source LLM framework, together with BentoML, addresses the challenges of deploying large language models by offering model switching, memory optimizations, multi‑GPU support, observability, and easy containerized deployment for production AI applications.
