A Visual Guide to Mixture of Experts (MoE) Architecture in Large Language Models

Introduction

When reviewing the latest large language models (LLMs), you often see the term “MoE” in the title. MoE stands for Mixture of Experts, a technique that improves LLM performance by using multiple specialized sub‑models, called experts, and a routing network that selects which experts process each token.

Core Components of MoE

Experts – each expert is a feed‑forward neural network (FFNN) attached to a transformer layer. During inference only a subset of experts is activated.

Router (Gate Network) – a learned network that decides, for every input token, which experts should receive the token.

Dense vs. Sparse Layers

In a standard transformer, the FFNN (dense layer) activates all parameters for every token. In a sparse MoE, only a few experts are activated, reducing computation while keeping model capacity high.

Expert Layers

Each expert layer is essentially a full‑connected FFNN. Because LLMs contain many decoder blocks, a token passes through multiple expert layers, forming a distinct execution path depending on the selected experts.

Router Layer

The router multiplies the input x by a weight matrix W, applies a Softmax to obtain a probability distribution G(x), and selects the top‑k experts (often top‑1). The selected experts’ outputs are multiplied by their gate values and summed to form the MoE layer output.

Load Balancing

Without explicit balancing, the router may repeatedly select the same expert, causing uneven expert utilization and training instability. Load‑balancing introduces an auxiliary loss that encourages a uniform distribution of token assignments across experts, measured by the coefficient of variation (CV) of expert importance scores.

Expert Capacity

Expert capacity limits the number of tokens an expert can process. When an expert reaches its capacity, additional tokens are routed to other experts. If all experts are full, the token “overflows” and skips the MoE layer, proceeding to the next network stage.

Case Study: Switch Transformer

Switch Transformer replaces the standard FFNN with a sparse MoE layer that uses a top‑1 routing strategy. It simplifies the architecture, improves training stability, and introduces a capacity factor that controls how many tokens each expert can handle.

Vision MoE (V‑MoE)

Vision Transformers (ViT) split images into patches that behave like language tokens. V‑MoE inserts sparse MoE layers into the ViT encoder, allowing many experts to process image patches while using batch‑priority routing to keep important patches from being dropped.

From Sparse to Soft MoE

Soft‑MoE replaces hard token‑to‑expert assignment with a continuous mixture. A learnable matrix Φ produces routing weights, which are softmax‑normalized to generate mixing coefficients. Tokens are then represented as weighted combinations of all experts, enabling gradient flow through all experts.

Case Study: Mixtral 8x7B

Mixtral 8x7B uses 8 experts each of size 5.6 B parameters. The full model stores 46.7 B parameters, but inference activates only 2 experts (≈12.8 B parameters), demonstrating the trade‑off between memory footprint and compute.

Conclusion

Mixture of Experts has become a foundational paradigm in modern deep learning. By selectively activating a subset of parameters, MoE enables scaling LLMs and vision models while keeping inference costs manageable. Understanding experts, routing, load balancing, and capacity is essential for designing efficient, high‑performing models.