Train-Free Iterative Pruning (GISP) Outperforms Layer-Wise LLM Compression

Background: Limitations of Uniform Layer-Wise Pruning

Uniform layer-wise pruning removes the same fraction of parameters from every layer (e.g., 25% per layer). This approach preserves each layer's output similarity to the original but ignores that layers differ in redundancy: some layers contain many unimportant parameters while others have few. The paper illustrates this with a three-column diagram: left column shows uniform 25% per-layer pruning with per-layer loss; middle column shows guided per-layer proportions (10%, 30%, 35%, 25%) with external guidance; right column shows global loss backpropagated to each layer with non-uniform proportions (15%, 45%, 25%, 15%). The uniform approach can miss small but task-relevant calibration signals.

GISP: Global Iterative Structured Pruning

GISP (Global Iterative Structured Pruning) prunes attention heads and MLP channels iteratively without any fine-tuning between steps. At each iteration, it computes importance scores for every head and channel, normalizes scores by block size, merges them into a global ranking, and prunes the lowest-scoring structures up to the current sparsity target. The pruned model becomes the input for the next iteration, yielding a nested sequence of subnetworks deployable at multiple sparsity levels.

Importance Scoring and Block-Wise Normalization

Importance is measured by the first-order Taylor term: absolute value of the gradient of the loss with respect to the parameter multiplied by the parameter value. Scores are summed within each head (for attention) or channel (for MLP). Raw head scores are an order of magnitude larger than channel scores; without normalization, global ranking would prune almost all heads and keep nearly all channels. Therefore, each block's total score is divided by its size (number of parameters) before global sorting. The paper reports that separate normalization for heads and channels empirically yields better accuracy.

Iterative Pruning Schedule

Given a total target sparsity and number of iterations, a linear schedule allocates increasing sparsity increments per iteration (later iterations prune more). Because each iteration recomputes gradients on the already-pruned model, previously removed structures cannot reappear. The authors note that 32 iterations (matching Llama2-7B's 32 layers) suffice to recover most of the perplexity gap for 50% sparsity: one-shot 50% pruning yields WikiText-2 perplexity 159.47, while 32-step iterative pruning reduces it to 64.07.

Experimental Results on Llama2-7B and Llama2-13B

Experiments use C4 for calibration and evaluate on WikiText-2 perplexity. Unpruned Llama2-7B perplexity is 12.19. At 40% sparsity, GISP achieves 34.54 vs. Wanda 51.85, FLAP 69.64, ShortGPT 189.17. At 50% sparsity, GISP achieves 64.07 vs. Wanda 81.47, FLAP 161.84, ShortGPT 387.94. For Llama2-13B (unpruned 10.98), GISP gives 26.56 at 40% and 42.07 at 50%. At 20% sparsity, all methods remain close to the unpruned baseline (GISP 17.01, Wanda 22.71). The paper also states Llama3-8B and Mistral-0.3-7B follow the same trend, with detailed numbers in later tables.

Ablation: Iterative vs. One-Shot Pruning

A figure shows perplexity vs. iteration steps (1, 32, 64, 128, 256) for 20%, 30%, 40%, 50% sparsity. The 50% curve spikes at step 1 then drops sharply by step 32 and plateaus. Another figure compares one-shot GISP (red line) vs. per-sparsity independent runs (blue dots) across sparsity levels 0.20–0.50 (log scale), with Wanda, OWL, FLAP, LLM-Pruner scattered above. The iterative approach consistently outperforms one-shot at high sparsity.

Importance Magnitude Analysis

A layer-wise plot shows average attention head importance (blue line) far above MLP channel importance (red line, near zero). An inset magnifies the channel importance scale. This confirms that without block-wise normalization, global ranking would discard heads almost exclusively.

Common Sense QA Evaluation

On common sense QA (average accuracy), unpruned Llama2-7B scores 66.68. At 40% sparsity, GISP scores 55.28 vs. Wanda 50.12; at 50%, GISP 48.54 vs. Wanda 43.52. Both methods degrade relative to the unpruned model, but GISP degrades less than layer-wise baselines.

Training-Free Design and Calibration

The authors explicitly avoid fine-tuning between iterations, citing computational cost and inconsistency with current large-model pruning practices. A separate table explores low-rank fine-tuning after pruning, but the core claim is that the iterative process itself requires no training. Calibration data differs per metric: C4 for WikiText-2 perplexity, and QA-specific data for common sense QA; results from different calibrations are not averaged.

Paper References

Paper: https://aclanthology.org/2026.acl-long.1653

Outstanding paper list: https://2026.aclweb.org/program/best_papers/

}

Code example

👀点击关注
AI前沿快速Follow