Can AI Truly Understand Your Photo Album? DeepImageSearch and the DISBench Benchmark

DeepImageSearch Paradigm

DeepImageSearch redefines image retrieval by abandoning the assumption that each photo can be evaluated independently. Instead, it treats a personal photo collection as a visual history and requires the system to perform corpus‑level contextual reasoning: locate relevant events, chain scattered visual clues, and return the target image(s) that satisfy a multi‑step query.

DISBench Benchmark

DISBench is the first benchmark designed for this paradigm. It contains:

57 users, nearly 110,000 photos (average timeline per user ≈ 3.4 years).

Two query categories: Intra‑Event (46.7 % of queries) – first identify the relevant event, then find the target within that event; Inter‑Event (53.3 % of queries) – require reasoning across multiple events in the album.

Each query points to an average of 3.84 target images.

The internal event structure is hidden from the model, forcing it to discover and exploit the latent event graph.

Dataset and leaderboard are hosted on HuggingFace:

https://huggingface.co/datasets/RUC-NLPIR/DISBench

https://huggingface.co/spaces/RUC-NLPIR/DISBench-Leaderboard

ImageSeeker Framework

ImageSeeker provides a systematic environment for evaluating agents on visual‑history search. It defines four essential capabilities:

Semantic Retrieval : natural‑language search over the whole album.

Spatio‑Temporal Filtering : apply precise time and location constraints.

Fine‑Grained Visual Confirmation : inspect candidate images and make detailed judgments.

External Knowledge Supplementation : incorporate encyclopedic facts required by the query.

To support long‑range reasoning, ImageSeeker implements a dual‑memory system:

Explicit State Memory : named subsets that persist intermediate results for later reuse.

Compressed Context Memory : when the dialogue window approaches its limit, the history is summarized into a global goal and a current action plan, preserving essential reasoning state.

Evaluation of Multimodal Agents

Using ImageSeeker, the authors evaluated a wide range of state‑of‑the‑art multimodal models, both closed‑source and open‑source:

GPT‑4o, GPT‑5.2, Gemini‑3‑Pro, Claude‑Opus‑4.5 (closed‑source)

Qwen‑VL‑235B/32B, GLM‑4.6V (open‑source)

Even the strongest model (Claude‑Opus‑4.5) achieved only ~29 % perfect‑answer rate on a single attempt. Embedding‑only baselines performed near random guessing because visually similar but context‑irrelevant images dominate the results.

Error analysis revealed two dominant failure modes:

Reasoning Errors (≈ 36‑50 % of errors): the model finds the correct clue but loses the plan, drops constraints, or stops prematurely.

Visual Discrimination Errors : confusing different views of the same object or mis‑identifying distinct objects.

Cross‑event (inter‑event) queries are the primary bottleneck: performance drops sharply compared with intra‑event queries, indicating that long‑range event linking is the hardest challenge.

Key Findings and Contributions

The work delivers three major advances:

A paradigm shift from isolated semantic matching to active, context‑aware reasoning over visual histories.

The DISBench benchmark, providing a high‑quality, large‑scale testbed for this new task.

The ImageSeeker framework, which uncovers critical weaknesses in current agents—particularly planning, state management, and long‑term inference—and offers a baseline for future research.

Relevant resources:

Paper: https://arxiv.org/abs/2602.10809

GitHub project: https://github.com/RUC-NLPIR/DeepImageSearch

Code example

收
藏
，
分
享
、
在
看
，
给
个
三
连
击呗！