Unlocking Trillion‑Parameter MoE Models: Expert Parallelism and Alibaba Cloud PAI‑EAS Deployment Guide

Background: Opportunities and Challenges of MoE Models

Mixture of Experts (MoE) has shown huge potential in large language models by using a "divide‑and‑conquer" approach: multiple expert sub‑networks are selected dynamically by a gating network, enabling efficient scaling to trillion‑parameter sizes while activating only a few experts per inference.

The key feature of MoE is sparse activation, allowing models like Kimi K2 (1.04 T parameters, 384 experts) to activate only 8+1 experts per inference, reducing active parameters to 32.6 B.

Sparse activation dramatically lowers compute and training costs for trillion‑scale models, but introduces new inference challenges: traditional tensor or pipeline parallelism suffers from low resource utilization, high communication overhead, and high cost.

Expert Parallelism (EP): The Best Partner for MoE Models

Expert Parallelism distributes different experts across multiple GPUs or machines, routing each request only to the required experts, thus avoiding the memory bottleneck of replicating all experts on every device.

Extreme memory optimization: breaks the single‑GPU memory limit, enabling deployment of hundred‑billion to trillion‑parameter MoE models on limited GPU clusters.

Ultra‑high performance: each expert runs independently on its own device, fully utilizing bandwidth and achieving high throughput.

Significant cost reduction: only the needed expert parameters are loaded, improving hardware utilization and lowering total cost of ownership.

Alibaba Cloud PAI‑EAS: Ready‑to‑Use Enterprise EP Solution

PAI‑EAS provides production‑grade EP deployment support, integrating PD separation, large‑scale EP, compute‑communication co‑optimization, and MTP into a unified optimization paradigm.

Key capabilities include:

EP custom deployment templates with pre‑built images, resource options, and run commands for one‑click deployment.

Aggregated service management with independent lifecycle control for Prefill, Decode, and LLM intelligent routing services.

Optimized EPLB to balance load from sparse activation and reduce expert migration overhead.

LLM intelligent routing for efficient PD separation and uniform resource usage.

Comprehensive monitoring (GPU, network, I/O, health checks) and self‑healing fault tolerance.

Flexible scaling policies, independent Prefill/Decode scaling, and gray‑release support.

Hands‑On: Deploy and Use DeepSeek‑R1 EP Service

Open the PAI console ( https://pai.console.aliyun.com/ ) and navigate to "Model Online Service (EAS)" → "Deploy Service" → "LLM Large Model Deployment".

Select the model "DeepSeek‑R1‑0528‑PAI‑optimized".

Choose the inference engine vLLM and the deployment template "EP+PD Separation‑PAI Optimized".

Configure resources for Prefill and Decode (e.g., ml.gu8tea.8.48xlarge or ml.gu8tef.8.46xlarge).

Adjust deployment parameters if needed (e.g., EP_SIZE, DP_SIZE, TP_SIZE).

Click "Deploy" and wait ~20 minutes for the EP service to become active.

After deployment, manage the service via the service list: view aggregated and sub‑service metrics, configure auto‑scaling, and perform online debugging using the "/v1/chat/completions" endpoint.

{
  "model": "",
  "messages": [
    {"role": "user", "content": "Hello!"}
  ],
  "max_tokens": 1024
}

Example curl request:

curl -X POST <EAS_ENDPOINT>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: <EAS_TOKEN>" \
    -d '{
        "model": "<model_name>",
        "messages": [
            {"role": "system", "content": "You are a helpful and harmless assistant."},
            {"role": "user", "content": [{"type": "text", "text": "hello"}]}
        ]
    }'

Python example:

import json, requests
EAS_ENDPOINT = "<EAS_ENDPOINT>"
EAS_TOKEN = "<EAS_TOKEN>"
url = f"{EAS_ENDPOINT}/v1/chat/completions"
headers = {"Content-Type": "application/json", "Authorization": EAS_TOKEN}
model = "<model_name>"
messages = [{"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "hello"}]
req = {"model": model, "messages": messages, "stream": True,
       "temperature": 0.0, "top_p": 0.5, "top_k": 10, "max_tokens": 300}
response = requests.post(url, json=req, headers=headers, stream=True)
if True:
    for chunk in response.iter_lines(chunk_size=8192, decode_unicode=False):
        msg = chunk.decode("utf-8")
        if msg.startswith("data"):
            info = msg[6:]
            if info == "[DONE]":
                break
            else:
                resp = json.loads(info)
                print(resp["choices"][0]["delta"]["content"], end="", flush=True)
else:
    resp = json.loads(response.text)
    print(resp["choices"][0]["message"]["content"])

After confirming correct responses, integrate the service into your production workflow.