Why Removing the Vision Encoder Improves Infrastructure Efficiency

1. Vision Encoder disrupts the regular computation graph

Pure‑text LLMs consist of a series of similar Transformer layers:

Tokens → Embedding → Transformer Layer 1 → ... → Transformer Layer N

This regularity eases cost modeling and parallel strategy composition (DP, TP, PP, CP, EP). Adding a Vision Encoder introduces a separate visual path:

Text IDs → Embedding ──────────┐
                              ├→ Merge → LLM → Loss
Pixels → ViT → Projector ──────┘

The Vision Transformer (ViT) processes image patches into visual tokens, and although both ViT and LLM use Transformer blocks, their depth, hidden size, sequence length, input shape, and optimal parallel settings differ, so a single execution strategy is sub‑optimal.

2. Multimodal samples have unpredictable compute cost

Dynamic‑resolution encoders cause the number of visual tokens to depend on image resolution, image count, and video frames. Example samples:

Sample A: pure text
Sample B: one 448×448 image
Sample C: eight high‑resolution images
Sample D: a long video

The cost of a sample can be expressed as:

Cost(sample) ≈ C_encoder(N_encoder_visual_tokens) + C_llm(N_text_tokens + N_llm_visual_tokens)

where N_encoder_visual_tokens is the number of patch/frame tokens fed to the encoder and N_llm_visual_tokens is the number of visual tokens after the projector that enter the LLM. This creates two load‑imbalance issues:

Different data‑parallel ranks may receive images of varying resolution, count, or video length, forcing the slowest rank to dictate synchronization.

Micro‑batches with differing visual load cause pipeline stage service times to fluctuate.

When training data mixes pure‑text, single‑image, multi‑image, and video samples, the execution path changes per sample, making simple batch‑size or token‑count statistics insufficient for workload modeling.

3. Vision Encoder and LLM often require different parallel strategies

Consider a common pipeline placement:

PP0: [Vision Encoder + Projector + LLM Layer 0..5]
PP1: [LLM Layer 6..17]
PP2: [LLM Layer 18..29]
PP3: [LLM Layer 30..41]

For pure‑text LLMs, layer counts per stage can be tuned so that stage runtimes are balanced. In the layout above, the time of PP0 is:

T_PP0 = T_encoder(images, resolution, frames) + T_projector + T_llm_layer_0..5

If the visual load varies, T_PP0 can dominate, causing downstream stages to idle and generate pipeline bubbles or back‑pressure.

Furthermore, large MoE LLMs often use high degrees of TP, CP, PP, and EP, while a few‑hundred‑million‑parameter Vision Encoder does not need such aggressive partitioning. Over‑splitting the encoder can shrink per‑GPU GEMM size and increase communication overhead; the optimal CP degree differs because of distinct sequence lengths and memory pressure; and ViT typically does not use MoE, so EP is unsuitable for it.

Two configuration options are compared:

Option A: Share a single parallel configuration for Encoder and LLM – simpler but may over‑split the Encoder or overload PP0.
Option B: Use separate parallel configurations – allows independent optimization, but requires extra data‑layout conversion at the boundary.

In short, a shared configuration can waste efficiency, while separate configurations increase scheduling and communication complexity.

4. What Encoder‑free solves and what it does not

Removing the deep ViT yields an "Encoder‑free" system where visual modeling shares the same decoder stack as text modeling. From an infrastructure viewpoint this brings three benefits:

Eliminates parallel‑configuration conflicts between Encoder and LLM.

Removes conditional branches that execute a deep Encoder only for certain samples.

Makes scheduling, profiling, and performance modeling align with existing LLM infrastructure.

However, varying image resolutions, counts, and video frame numbers still produce different numbers of visual tokens, so sample‑level sequence length and load variance are not fully eliminated; the improvement is that these tokens no longer pass through a separate, heavily parallelized ViT.