DeepImageSearch Ushers in the Deep Search Era: Enabling AI to Understand Visual Histories

1. From Isolated Matching to Understanding Life Stories

DeepImageSearch proposes a fundamentally new image‑retrieval paradigm: instead of treating each photo as an independent island and performing one‑by‑one semantic matching, the system performs corpus‑level contextual reasoning across a user's visual history. The core insight is that true album search requires planning a search path, linking scattered clues, and constructing an evidence chain, much like a detective solving a case.

The authors illustrate the limitation of traditional retrieval with a music‑festival example. To find a photo of “the lead singer standing alone on stage” among thousands of similar concert shots, a user remembers a blue‑white logo on the venue banner. Traditional models cannot use that clue because the target image itself lacks distinctive visual features. DeepImageSearch must first locate the logo‑containing photo, anchor the event, and then retrieve the specific singer‑only shot, turning a single‑step match into a multi‑step exploration.

2. DISBench – A Challenging Benchmark for Deep Image Search

To drive research on this new paradigm, the authors built DISBench (DeepImageSearch‑Bench), a high‑difficulty benchmark that evaluates two query types:

Intra‑Event queries (46.7%) : locate the target within a single event after first identifying the event (e.g., find the singer‑only photo after anchoring the festival via the logo).

Inter‑Event queries (53.3%) : discover relations across multiple events (e.g., find all photos of a specific statue that appears in two trips six months apart).

DISBench covers 57 users, nearly 110 000 photos, with an average visual‑history span of 3.4 years per user. Each query points to an average of 3.84 target images. The benchmark is released on HuggingFace (dataset and leaderboard) and the paper is available on arXiv (https://arxiv.org/abs/2602.10809).

3. ImageSeeker – Probing the Capabilities Required for Visual‑History Search

The ImageSeeker framework is designed to explore what abilities an agent needs for deep visual‑history search. The authors identify four essential capabilities:

Semantic retrieval – natural‑language search over the album.

Spatio‑temporal filtering – handling time and location constraints.

Visual confirmation – fine‑grained inspection of candidate photos.

External knowledge – answering encyclopedic aspects of a query.

These capabilities can be combined: an agent can save intermediate results as named subsets, then continue searching within a subset, enabling multi‑step reasoning with progressive narrowing.

Because a single query may require dozens of interaction steps, ImageSeeker introduces a dual‑memory mechanism: an explicit state memory (named subsets) and a compressed context memory that summarizes the global goal and current plan when the dialogue history approaches the model’s context limit.

4. Evaluation – Even the Strongest Multimodal Models Fall Short

The authors plugged several leading multimodal models into ImageSeeker, including closed‑source GPT‑4o, GPT‑5.2, Gemini‑3‑Flash/Pro, Claude‑Opus‑4.5, and open‑source Qwen3‑VL‑235B/32B and GLM‑4.6V. Across DISBench, the best model (Claude‑Opus‑4.5) achieved only ~29% perfect‑rate per attempt; the top open‑source model (GLM‑4.6V) scored less than 40% of the best closed‑source score. Traditional embedding‑only retrieval performed near random because visually similar photos overwhelm the signal.

Failure analysis shows that the dominant error type (36‑50% of errors) is reasoning failure: models find the correct clue but lose it during multi‑step planning, drop constraints, or stop prematurely. Visual discrimination errors are secondary. Further findings include:

Cross‑event reasoning is the main bottleneck : models perform noticeably better on intra‑event queries than on inter‑event ones.

Better retrieval does not guarantee better answers : scaling up embedding models yields inconsistent gains, indicating that the core challenge lies in reasoning over retrieved results.

Potential for improvement : ensemble strategies such as Best@k and majority voting raise overall scores, suggesting that models have latent correct reasoning that can be unlocked with better orchestration.

5. Conclusion

DeepImageSearch defines a shift from passive semantic matching to active contextual reasoning in image retrieval. DISBench provides the first rigorous benchmark for this capability, and ImageSeeker offers a systematic exploration of the required agent abilities. The study reveals that current state‑of‑the‑art multimodal models are limited more by planning and long‑range reasoning than by visual perception, highlighting a clear research direction for future AI‑driven visual‑history search.

Project resources: GitHub repository (https://github.com/RUC-NLPIR/DeepImageSearch), HuggingFace dataset and leaderboard links, and the original arXiv paper.