VisionHOPE: Self-Modifying Visual Backbone Learns How to Learn Per Image

Human vision adapts its processing strategy while scanning an image: early observations inform how later regions are interpreted. VisionHOPE brings this capability to neural networks by making the learning rules themselves — feature association, update step size, memory retention — a function of the content already processed.

Background: From Fixed Rules to Adaptive Learning

Traditional visual backbones fix their learning dynamics at training time:

CNNs use static convolution kernels; long-range dependencies require many layers.

Vision Transformers (ViT) enable global interaction via self-attention, but projection parameters remain fixed and computation scales quadratically with token count.

State-space models (Mamba) achieve linear complexity through recurrent state transitions, yet the input-to-state mapping is still trained offline.

Test-Time Training (TTT) inserts a small learner that updates its parameters during inference, but the meta-rules governing that update (learning rate, memory decay) are predetermined.

VisionHOPE extends this trajectory by letting the image itself modulate the meta-rules during a single forward pass.

Core Architecture: SRNL and Five Coupled Memories

The Self-Referential Nested Learning (SRNL) module maintains five differentiable memory banks that are updated sequentially as tokens arrive:

Content memory — stores visual features for readout.

Key memory — generates write indices to associate new inputs with existing content.

Value memory — produces the internal learning target, steering the direction of correction.

Learning-rate memory — controls the magnitude of each update.

Retention-rate memory — determines how much old information is preserved.

At each step, the current memories emit Key, Value, learning-rate, and retention-rate vectors. A Delta Gradient Descent (DGD) write mechanism computes the discrepancy between the current Key response and the new feature, writing only the residual to reduce redundant accumulation (e.g., repeated sky textures). Updated memories then participate in processing the next token.

Stability Guarantees

Self-referential updates risk runaway amplification. The paper introduces two complementary constraints:

Soft injection ceiling — smoothly limits the injected feedback strength; higher retention reduces allowable new input, keeping the sum bounded.

Spectral truncation — caps the amplification of existing memories during propagation, activating only when the candidate step exceeds a proven stability threshold.

Theoretical analysis proves that under both constraints, the norms of all five memories remain bounded by their initial values throughout the scan, for both token-wise and chunk-wise recurrence.

2D Adaptation: Four-Direction Scanning and Block Parallelism

Images are 2D; SRNL is inherently sequential. VisionHOPE scans along four raster orders (left→right, right→left, top→bottom, bottom→top), each with independent memory sets. Outputs are spatially realigned and fused by channel-wise weighting. For efficiency, rows/columns are processed in chunks (e.g., 14 tokens): within a chunk, intermediate quantities are computed in parallel from the chunk-start state, then updates are accumulated sequentially. On P-VisionHOPE-S, chunk size 14 yields 2.86× speedup over token-wise inference with only 0.1% Top-1 drop (82.3% vs 82.4%).

Experimental Results

ImageNet-1K Classification (224×224)

VisionHOPE-T: 83.7% Top-1, 5.6M params — beats VMamba-T (82.2%, 9.0M) by 1.5 pp with 3.4M fewer params.

VisionHOPE-S: 85.2% Top-1, 52.9M params — beats VMamba-B (83.9%, 89M) by 1.3 pp at 60% param count.

VisionHOPE-B: 85.6% Top-1.

Across CNN, ViT, SSM, and TTT families, VisionHOPE matches or exceeds all counterparts at each scale.

Downstream Tasks

COCO detection/segmentation (Mask R-CNN, 1× schedule) : all three scales achieve best or tied-best box AP and mask AP.

ADE20K semantic segmentation (UPerNet) : likewise best or tied-best mIoU.

Gains transfer from classification to spatial tasks, confirming general-purpose backbone utility.

Qualitative Analysis

Effective receptive field (block 2, 512×512 input) : P-VisionHOPE-S covers a broad 2D region with a sharp center peak, unlike ResNet-50 (local), DeiT-S/TTT-ViT³-S (center-biased), or Vim-S (directional).

Spatial learning-rate map : larger step sizes align with foreground objects and salient structures; backgrounds receive smaller steps — direct evidence that image content modulates update intensity.

Open-Source Release

All code, CUDA kernels, and pre-trained weights are MIT-licensed at https://github.com/PSRben/VisionHOPE. Four integration levels are provided:

SRNL — sequence module for [B, N, D] tensors.

VisionHOPEOperator — 2D operator for [B, C, H, W] feature maps.

VisionHOPEBlock — full block with local conv, FFN, residual.

create_model() — ready-to-use classification backbones (Tiny/Small/Base).

All modules preserve tensor shapes and support PyTorch autograd. Installation requires Linux, NVIDIA GPU, CUDA 12.1, Python 3.10; kernels JIT-compile on first call. Weights download via huggingface_hub; prepare_for_inference() fuses parameters for deployment (keep an unfused copy for further training).

Future Directions

Domain-specific validation : medical imaging, remote sensing, industrial inspection — where subtle anomalies hide in clutter — may benefit from content-adaptive feature association and step sizing.

Temporal extension : current memories reset per image. For long video, autonomous driving, and embodied AI, the team envisions carrying memories across frames, requiring solutions for scene-change reset and error propagation.

Cross-modal memory : SRNL's stability-guaranteed self-modification offers a substrate for joint vision-language memory and long-context language modeling.

VisionHOPE demonstrates that a visual backbone can rewrite its own learning rules on the fly, grounded in provable stability and verified by strong empirical gains. The open release invites the community to test how far this self-modifying paradigm can scale.