How Knowledge‑Guided Context Optimization Boosts Zero‑Shot Vision‑Language Models

Why CoOp Fails on Unseen Classes

Prompt tuning via Context Optimization (CoOp) replaces fixed hand‑crafted templates (e.g., "a photo of a [Class]") with learnable context vectors. This boosts accuracy on base (seen) categories but causes catastrophic forgetting on novel categories. Experiments on 11 benchmarks show that CoOp raises base accuracy yet lowers new‑class performance below the original zero‑shot CLIP baseline.

Learned prompts that drift farther from hand‑crafted prompts lead to larger performance degradation on unseen classes.

Geometry of Forgetting

The Euclidean distance between the learned prompt embedding w_coop and the original CLIP prompt embedding w_clip correlates directly with the drop in new‑class accuracy. Larger distances indicate more severe forgetting.

Knowledge‑Guided Context Optimization (KgCoOp)

KgCoOp introduces a regularization term that penalizes the Euclidean distance between the fine‑tuned prompt and the original CLIP prompt, encouraging the model to retain its generic knowledge while adapting to a downstream task.

The training objective combines the standard cross‑entropy loss L_ce with the knowledge‑guided loss L_kg: L = L_ce + λ·L_kg where L_kg = ||w_coop – w_clip||_2^2 measures the squared Euclidean distance between the learned prompt embedding w_coop and the CLIP anchor w_clip. The hyper‑parameter λ controls the strength of the constraint.

Experimental Setup and Benchmarks

KgCoOp is evaluated on 11 diverse image‑classification datasets (ImageNet, Caltech‑101, Oxford‑Pets, Stanford‑Cars, Flowers‑102, Food‑101, FGVCAircraft, EuroSAT, UCF‑101, DTD, SUN‑397) using ResNet‑50 and ViT‑B/16 backbones with a 16‑shot setting.

Results: Closing the Base‑to‑New Gap

KgCoOp achieves the highest harmonic mean across all settings. Compared with the CoOp baseline, KgCoOp improves new‑class accuracy by 5.61 % (ViT‑B/16, 16‑shot) and raises the overall harmonic mean to 77.0 %, surpassing CoCoOp (75.83 %) and ProGrad (76.16 %). The method also attains the best new‑class scores on specialized datasets such as EuroSAT and UCF‑101.

Domain Generalization

In a domain‑generalization scenario, models are first fine‑tuned on 16‑shot ImageNet and then evaluated on out‑of‑distribution variants (ImageNet‑V2, Sketch, A, R). KgCoOp consistently outperforms baselines, demonstrating robust transfer to shifted data distributions.

Efficiency and Overhead

Computing the Euclidean distance adds negligible overhead. KgCoOp processes images at ~6 M samples/s, far faster than CoCoOp (≈26× slower) and ProGrad (≈22 ms/image). The L_kg term can be plugged into other prompt‑tuning frameworks, yielding consistent gains.

Sensitivity to λ and Context Length

Increasing λ reduces the prompt‑anchor distance and improves harmonic mean up to λ≈8.0; larger λ over‑constrains the model, hurting base performance. Extending the context length from M=4 to M=8 tokens yields additional gains on both base and new classes when compute permits.

Limitations

KgCoOp still faces a trade‑off: stronger knowledge‑guided constraints improve unseen accuracy but can slightly lower base accuracy. The additional hyper‑parameter λ introduces tuning complexity, and an overly tight constraint may hinder adaptation to highly specialized downstream tasks.

Conclusion

KgCoOp offers a lightweight, geometry‑based regularization that preserves the generic knowledge of large‑scale vision‑language models while enhancing their ability to generalize to novel categories. By minimizing a simple Euclidean distance loss, it provides a practical tuning recipe that can be integrated into existing prompt‑learning pipelines with minimal computational cost.