Running a 35B MoE Model at 32 tokens/s on $600 Tesla P100 GPUs

Hardware and environment

Dell R730XD server equipped with two Tesla P100 GPUs (16 GB each). An optional NVLink bridge can be installed for higher cross‑GPU bandwidth. The host runs ESXi 8.0, the guest OS is AlmaLinux 10.2 (RHEL compatible). Docker + Ollama provides the LLM inference service on port 11434.

ESXi host and VM configuration

Enable PCI passthrough

Log in to the ESXi UI ( https://<ESXi_IP>/ui) as root, navigate to Manage → Hardware → PCI Devices , select both Tesla P100 cards, enable passthrough, and reboot the host.

VM hardware settings

Create a VM and add the two GPUs as PCI devices. In the memory options enable Reserve all guest memory . Set the boot firmware to UEFI and disable Secure Boot.

Advanced VM parameters

pciPassthru.use64bitMMIO = TRUE          # Enable 64‑bit MMIO for large‑memory GPUs
pciPassthru.64bitMMIOSizeGB = 64          # Address space for two 16 GB GPUs (64 GB sufficient)
hypervisor.cpuid.v0 = FALSE            # Hide virtualization flag
latency.sensitivity = high               # Optimize scheduling for latency‑sensitive workloads

VMs with PCI passthrough cannot use snapshots, suspend, or vMotion.

AlmaLinux 10 initialization and kernel upgrade

sudo dnf install -y epel-release
sudo dnf config-manager --set-enabled crb
sudo dnf update -y
sudo reboot

Disable the Nouveau open‑source driver

# Install build dependencies
sudo dnf install -y gcc make kernel-devel-$(uname -r) kernel-headers-$(uname -r) elfutils-libelf-devel dkms
# Blacklist Nouveau
cat <<EOF | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0
EOF
# Regenerate initramfs and reboot
sudo dracut --force
sudo reboot

Verify with lsmod | grep nouveau; no output indicates success.

Install NVIDIA official driver (v580.105.08)

cd /tmp
curl -L -o NVIDIA-Linux-x86_64-580.105.08.run \
  https://us.download.nvidia.com/tesla/580.105.08/NVIDIA-Linux-x86_64-580.105.08.run
chmod +x NVIDIA-Linux-x86_64-580.105.08.run
sudo ./NVIDIA-Linux-x86_64-580.105.08.run --no-opengl-files --no-x-check --dkms -s

Confirm both GPUs appear with nvidia-smi.

Enable GPU persistence mode

sudo nvidia-smi -pm 1
# Add to crontab for boot‑time and hourly refresh
(echo "@reboot /usr/bin/nvidia-smi -pm 1"; echo "0 * * * * /usr/bin/nvidia-smi -pm 1") | crontab -

Install NVIDIA Container Toolkit

curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Deploy Ollama with dual‑GPU support

Create directory

mkdir -p /opt/ollama && cd /opt/ollama

docker‑compose.yml

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - /opt/ollama/data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
      - OLLAMA_KEEP_ALIVE=-1
      - OLLAMA_MAX_LOADED_MODELS=1
      - OLLAMA_NUM_PARALLEL=1
      - OLLAMA_FLASH_ATTENTION=1
      - OLLAMA_KV_CACHE_TYPE=q4_0
      - OLLAMA_TIMEOUT=600
      - OLLAMA_CONTEXT_LENGTH=16384
      - OLLAMA_SCHED_SPREAD=1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    shm_size: '16gb'
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "3"

Start service

docker compose up -d

The first run pulls the ~1‑2 GB Ollama image.

Model pull, run, and monitoring

# Pull 35 B model (≈23 GB)
 docker exec ollama ollama pull qwen3.6:35b
# Run a test prompt
 docker exec ollama ollama run qwen3.6:35b "请简要介绍一下你自己"
# List loaded models
 docker exec ollama ollama list
# Monitor GPU usage per second
 watch -n 1 nvidia-smi

Both GPUs should show memory usage, confirming cross‑GPU tensor splitting.

Performance comparison

qwen2.5‑coder:14b (dense) – eval rate 16.68 tokens/s, prompt eval 9.34 tokens/s, model size ≈13 GB, active parameters 140 B.

qwen3.6:27b (dense) – eval rate 10.41 tokens/s, prompt eval 0.39 tokens/s, model size ≈17 GB, active parameters 278 B.

qwen3.6:35b (MoE) – eval rate 32.77 tokens/s , prompt eval 195.37 tokens/s , model size ≈23 GB, active parameters ≈120 B (≈1/3 of total 360 B).

The 35 B MoE model activates only about one‑third of its parameters per token, making it the fastest despite its larger total size.

NVLink (optional)

If an NVLink bridge is installed, cross‑GPU bandwidth is higher than PCIe. Check status with:

nvidia-smi nvlink --status
nvidia-smi topo -m

NVLink appears as "NVLink" in the topology matrix.

Firewall configuration

sudo firewall-cmd --permanent --add-port=11434/tcp
sudo firewall-cmd --reload

Environment summary

Host: kernel upgraded, NVIDIA driver 580.105.08 installed, GPU persistence enabled via crontab, NVIDIA Container Toolkit configured, Docker runtime set to NVIDIA.

Container: Ollama latest image, OLLAMA_SCHED_SPREAD=1 forces even distribution across the two GPUs, qwen3.6:35b model loaded.

Conclusion

With two $600 Tesla P100 cards in a Dell R730XD, the qwen3.6:35b MoE model runs at over 32 tokens/s, demonstrating that legacy GPUs can deliver practical LLM inference when correctly virtualized and configured.