From Dense to MoE: Decoding Total vs. Activated Parameters

This article starts from dense and sparse models, then examines experts and routing mechanisms in MoE. Understanding the difference between total and activated parameters makes it easier to judge how much memory an MoE model needs and what bottlenecks it faces during prefill and decode. Before reading, you may want to review "Large Model Inference Process Explained" to understand model loading, prefill, KV cache, and decoding basics.

Introduction

Previously, judging a large model's memory footprint and speed relied mainly on parameter count. More parameters generally meant larger weights and more computation per inference. For dense models this holds: a 27B dense model uses all its main-layer parameters for every token, so total parameters reflect both weight size and compute pressure.

In MoE models, however, two numbers appear: total parameters and activated parameters. A model may have hundreds of billions of total parameters but only activate a few billion per token. Relying solely on the total parameter count in the model name no longer reveals per-token compute.

Dense Models

Dense models follow a fixed computation path. Every token passes through all main model layers (attention and feed-forward networks). Regardless of input content, the same major computational structures are traversed.

Input Token
  ↓
Through each attention and feed-forward layer
  ↓
Main parameters participate in computation
  ↓
Output next token prediction

Ignoring attention and other overhead, the forward compute for a dense model can be approximated as:

Compute ≈ 2 × Model Parameters × Token Count

Here one multiply-add counts as 2 FLOPs. This is not a precise performance formula but illustrates the basic relationship: as total parameters grow, both model weight size and per-token compute typically increase together.

Sparse Models

Sparse models can have many parameters but do not necessarily engage all of them for a single input. Sparsity can appear in different forms:

Weight sparsity: Part of weights are pruned, set to zero, or skipped.

Activation sparsity: Only a subset of neurons or modules are activated per computation.

Structured sparsity: A fixed structure selects a portion of parameters to participate according to a rule.

The goal is to retain large model capacity while reducing actual compute per inference. Note that GPU "Sparse TFLOPS" refers to hardware theoretical throughput under specific sparse structures, which is different from MoE's model-level expert selection.

How MoE Works

MoE (Mixture of Experts) is a representative structured sparse model. In many MoE models, parts of the Transformer feed-forward networks are replaced by multiple "experts." Each expert has its own parameters, but a token only passes through a few experts selected by a router.

Input Token
  ↓
Shared attention layers, etc.
  ↓
Router decides which experts to use
  ↓
Only Top-K experts activated
  ↓
Merge expert outputs, continue computation

Think of it as a company: a dense model involves all major departments for every task; MoE adds a dispatcher that routes tasks to a few specialist departments. However, shared structures (attention, normalization, router) still participate every time; only the expert portion is conditionally selected.

Total Parameters vs. Activated Parameters

Understanding MoE starts with separating total parameters from activated parameters.

Total parameters: Sum of all shared parameters and all expert parameters. Determines model weight size and memory capacity requirements.

Activated parameters: Parameters actually participating in computation for a single token. Determines per-token primary compute and weight reads.

Non-activated parameters: Expert parameters not selected for the current token. They do not participate in this round's computation but remain part of the model weights.

For example, a MoE model with 100B total parameters but 10B activated per token still stores the full 100B in its model files; only a subset is invoked per token based on routing.

To judge if a model fits in memory, look at total parameters and quantization precision. To estimate per-token compute, look at activated parameters. This is an approximation; real inference also involves shared parameters, attention compute, KV cache, expert routing, and framework efficiency.

MoE in the Full Inference Process

MoE still goes through model loading, prefill, and decode, but parameter access patterns change.

1. Model Loading: Usually Load All Parameters

Loading typically places both shared parameters and all expert parameters into VRAM or unified memory. The router cannot know in advance which experts future tokens will need; loading only currently used experts would cause frequent data transfers from system memory or disk, increasing latency. Thus a 100B total / 10B activated MoE model still demands memory for 100B parameters, even though per-token compute resembles a smaller model. If memory is insufficient, some frameworks can keep part of the experts in system memory and transfer on demand, but expert switching and data movement usually slow things down.

2. Prefill: Each Token Activates Few Experts

During prefill, many prompt tokens are processed in parallel. The router selects a few experts per token, so per-token compute aligns with activated parameters, not total parameters. However, different tokens in a prompt may be routed to different experts. For a 2000-token prompt, while each token only uses a few experts, the aggregate may involve many or all experts. This doesn't turn each token into full-parameter compute, but makes expert scheduling, data organization, and load balancing more complex.

3. Decode: Read Shared Parameters and Selected Expert Parameters

In the token-by-token decode phase, each step generates one new token. The router selects a few experts, and the GPU primarily reads and computes:

Shared parameters (attention layers, etc.)

Router parameters

Currently selected expert parameters

Ever-growing KV cache

If the model fully resides in VRAM, all experts are already on GPU; decode simply reads shared weights and the chosen expert weights for each token, not reloading from disk every step. Therefore, the common dense-model speed estimate:

Theoretical generation speed ≈ Memory Bandwidth ÷ Total Model Weight Size

cannot be directly applied to MoE. For MoE, per-token primary weight reads approximate:

Per-token primary weight reads
≈ Shared layer weights + Currently activated expert weights

This remains an approximation; expert switching, cache hits, batch size, routing results, and framework implementation all affect measured speed.

What to Watch in Each Stage

Model loading: Loads shared + all experts → focus on total parameters, quantization, memory capacity.

Prefill: Each token activates few experts; many tokens may cover many experts → focus on activated parameters, GPU compute, expert scheduling.

Decode: Each step reads shared + selected expert parameters → focus on activated weight reads, memory bandwidth, framework efficiency.

Estimation Example: 100B MoE Model

Assume a MoE model with 100B total parameters, 10B activated per token, FP16 weights (2 bytes per parameter). Loading the model requires storing all weights: 100B total parameters × 2 bytes ≈ 200 GB But per-token primary compute involves ~10B activated parameters. For a 2000-token prefill, primary compute can be roughly estimated as:

Prefill primary compute ≈ 2 × Activated parameters × Input tokens
≈ 2 × 10B × 2000
≈ 40 trillion FLOPs

These numbers are not for predicting real speed; they illustrate two concepts: 200 GB reflects model loading pressure, 40 trillion FLOPs reflects the prompt's primary compute. Actual results are also affected by shared layers, attention, expert routing, and framework efficiency.

Conclusion

When encountering an MoE model, break "how many B" into concrete questions: How large is the model file? How much memory to load it? How many parameters activate per token? Which shared and expert weights are read during generation? Can the current inference framework schedule experts efficiently?

In dense models, total parameters usually reflect both model size and compute pressure. In MoE, these two aspects are separated: all parameters determine overall capacity, while each token only calls a subset for computation.