Can Evolutionary Algorithms Auto-Design Training-Free Vision-Language Model Adaptations?

Background

Pre‑trained vision‑language models (VLMs) have become the backbone of many computer‑vision tasks, especially few‑shot image classification. To adapt a VLM to a downstream task without additional training, researchers typically employ prompt tuning, adapters, or other lightweight modules. Existing adaptation techniques are handcrafted, which requires substantial expert effort and limits scalability.

EvoVLMA Overview

The proposed Evolutionary Vision‑Language Model Adaptation (EvoVLMA) automatically discovers efficient, training‑free adaptation pipelines for VLMs. The authors first isolate two essential functional modules that determine the performance of a training‑free adapter:

Feature selection : decides which visual or textual features are passed to the downstream classifier.

Logits computation : defines how the selected features are transformed into class logits.

EvoVLMA searches for optimal implementations of these modules using a two‑stage evolutionary algorithm assisted by large language models (LLMs) . The search proceeds in a “divide‑and‑conquer” fashion, optimizing the feature‑selection stage first and then the logits‑computation stage, thereby reducing the combinatorial explosion of the joint search space.

Stage 1 – Evolution of Feature Selection

Each individual in the population encodes a candidate feature‑selection strategy (e.g., a subset of transformer layers, pooling operations, or learned linear projections). The evolutionary loop performs:

Mutation: random alteration of the encoded strategy (add/remove a layer, change pooling type).

Crossover: recombination of two parent encodings.

Evaluation: the candidate code is sent to an LLM, which expands it into executable Python (or PyTorch) snippets. The snippet is then run on a lightweight validation set to obtain a proxy accuracy.

Stage 2 – Evolution of Logits Computation

After fixing the best feature‑selection module, EvoVLMA evolves the logits‑computation block. Candidates may include different normalization schemes, temperature scaling, or linear classifiers. The same mutation‑crossover‑evaluation pipeline is applied, with the LLM generating the corresponding code.

Search‑Efficiency Enhancements

To keep the evolutionary search tractable, EvoVLMA incorporates three engineering mechanisms:

Low‑precision code conversion : generated code is automatically cast to float16 (or bfloat16) where possible, reducing memory and speeding up inference.

Web‑based code execution sandbox : each candidate snippet is executed in an isolated container accessed via a lightweight HTTP API, eliminating the overhead of launching full training environments.

Process monitoring : a watchdog tracks execution time, CPU/GPU usage, and crashes; any run exceeding a preset budget is terminated and penalized in the fitness score.

Experimental Evaluation

The method was evaluated on standard few‑shot image‑classification benchmarks (e.g., 8‑shot CIFAR‑FS and mini‑ImageNet). EvoVLMA‑discovered adapters consistently outperformed manually designed baselines such as APE, CoOp, and CLIP‑Adapter. In the 8‑shot setting, the EvoVLMA‑enhanced APE algorithm achieved an absolute gain of 1.91 percentage points in top‑1 accuracy compared with the original APE implementation. Additional ablations demonstrated that:

Stage‑wise evolution yields higher final accuracy than joint evolution.

Low‑precision conversion reduces average evaluation time by ~40 % with negligible loss in performance.

LLM‑generated code improves the diversity of candidate solutions, leading to faster convergence.

Implementation and Reproducibility

The full EvoVLMA framework, including the evolutionary engine, LLM prompting templates, and the web‑execution backend, is released under an open‑source license.

git clone https://github.com/kding1225/EvoVLMA.git

All experiments can be reproduced by following the instructions in the repository’s README.md.