Deploying DeepSeek LLMs On-Premises: Step‑by‑Step Guide and Hardware Sizing

This article presents a detailed technical analysis of privately deploying DeepSeek large language models (LLMs) in enterprise environments.

1. Deployment Process Overview

The typical workflow for deploying a DeepSeek‑R1 model on a bare‑metal server includes:

Prepare the software stack (OS, drivers, firmware).

Install Ascend NPU firmware and drivers.

Install supporting software packages such as MindSpore (inference engine), CANN (heterogeneous computing architecture), and related libraries.

Download the model weights (e.g., 671B full‑precision or 70B distilled versions) and convert them to the desired precision (FP8 or FP16).

Deploy the inference service (configure environment variables, start containers, verify functionality).

Perform performance tuning of the inference engine and related software.

Set up network security, logging, and monitoring dashboards.

2. Model and Runtime Parameter Considerations

Choosing appropriate model parameters (size, precision) and runtime settings (context length, batch size) directly influences the required compute resources. Complex scenarios such as financial analysis, medical imaging, or legal document processing may need models larger than 70B parameters with a context length of 32K tokens, while simpler internal knowledge‑base or chatbot use cases can work with smaller configurations.

3. Hardware Sizing Calculations

Memory requirements are calculated as:

Model parameters × precision (e.g., 70 B × 1 byte for FP8 = 70 GB).

Activation cache : model parameters × precision × dynamic factor (0.1‑0.5). For a 70B model at FP8 with a factor of 0.25, the activation cache is ≈ 17.5 GB.

Output tensor cache : batch size × sequence length × vocab size × precision ÷ 1024³. With batch = 16, sequence = 8192, vocab = 128 256, precision = 1 byte, the output cache ≈ 15.66 GB.

Fixed overhead for the AI accelerator and software stack ≈ 1 GB.

Summing these components yields a total memory requirement of roughly 104 GB for a typical production deployment of a 70B model.

Hardware options that satisfy this memory demand include:

NVIDIA H200 (141 GB VRAM) or H20 (96 GB VRAM).

Huawei Ascend 910B (64 GB VRAM) – multiple cards may be needed to reach the required capacity.

When selecting hardware, consider not only memory capacity but also compute performance, memory bandwidth, and inter‑connect bandwidth, as they determine inference throughput and efficiency.

4. Software Stack and Ecosystem Support

The AI accelerator’s firmware and driver control low‑level computation efficiency. For Huawei Ascend, the firmware manages power, OS, and chip control, while the driver enables interaction with the CANN runtime.

Key software packages that facilitate efficient model deployment include:

Heterogeneous Computing Architecture (CANN) : Provides a unified programming model for CPU, GPU, and NPU.

Inference Engine (MindSpore/MindIE) : Optimizes model execution, supports quantization (FP32→INT8), operator fusion, and memory reuse.

Collective Communication Library (HCCL/NCCL) : Enables high‑performance multi‑card and multi‑node communication for data and model parallelism.

Infrastructure Management Platform (DCS, DGX SuperPOD, etc.) : Offers resource virtualization, elastic scaling, operation & maintenance, and disaster‑recovery capabilities for AI workloads.

These components together determine the overall utilization efficiency of the compute hardware and the ease of scaling AI services.

5. Performance and Cost Trade‑offs

Beyond meeting the minimum memory requirement, enterprises should evaluate inference speed (tokens / second) and concurrency needs. The choice of accelerator count, compute capability, memory bandwidth, and inter‑connect bandwidth all affect the cost‑performance balance.

For DeepSeek‑R1‑70B, the recommended configuration in a production environment is a 512 GB memory pool, which can be achieved with eight 64 GB Ascend 910B cards or equivalent NVIDIA solutions.

6. Security and Monitoring

Deployments should include network security hardening, log management, and monitoring dashboards to ensure reliable operation and compliance with enterprise policies.

Overall, successful private LLM deployment requires careful alignment of model specifications, hardware resources, software stack compatibility, and operational safeguards.