AgentHOI: Training‑Free Human‑Object Interaction Detection with Multimodal LLMs

Human‑Object Interaction (HOI) detection traditionally relies on supervised learning with a fixed vocabulary and extensive annotation, which limits scalability and generalization to open‑world scenarios. The authors identify two main drawbacks: costly exhaustive labeling and poor transfer to unseen action‑object combinations.

AgentHOI Framework

AgentHOI reframes HOI detection as a two‑stage process: (1) context‑aware multi‑round reasoning (CMR) to infer possible "action‑object" pairs, and (2) multifaceted interaction localization (MIL) to precisely locate the involved person and object. The base system (AgentHOI‑Base) combines two off‑the‑shelf visual modules: GPT‑4o for multimodal understanding and reasoning, and GroundingDINO for spatial grounding. No HOI‑specific training data or parameter updates are required.

Context‑aware Multi‑round Reasoning

Round 1 – Initial HOI identification: the multimodal LLM scans the whole image and extracts the most salient interaction (e.g., "person holding a bottle").

Round 2 – HOI mining: the previous result is fed back as context, prompting the model to search for additional interactions, reducing missed pairs in crowded scenes.

Round 3 – Action re‑allocation: for each detected pair, the model considers alternative fine‑grained verbs (e.g., "grasp" vs. "hold") and augments the label set.

This three‑stage pipeline discovers dominant interactions, fills omissions, and refines verb granularity.

Multifaceted Interaction Localization

After obtaining interaction triples, AgentHOI generates detailed textual descriptors for both the person and the object, incorporating semantic, spatial, and appearance cues. For example, the person description may become "the left‑side individual wearing a black shirt and a hat, holding or grasping the bottle". GroundingDINO then searches the image for regions matching each description, outputting bounding boxes and mitigating confusion among similar instances.

Experimental Evaluation

AgentHOI is evaluated on three benchmarks:

HICO‑DET (9,658 images, 600 HOI combos). In zero‑shot settings, AgentHOI achieves 29.85 mAP on unseen verbs (UV) and 38.64 mAP on rare‑unseen combos (RF‑UC), surpassing all listed baselines.

Robustness tests with style transfer and image degradation (blur, noise, compression) show that supervised methods suffer notable drops, while AgentHOI maintains leading performance across UV, UO, and RF‑UC configurations.

SWIG‑HOI (larger vocabulary). Without any fine‑tuning, AgentHOI reaches 13.00 mAP overall and 11.61 mAP on unseen categories, exceeding the fully supervised CMD‑SE (10.70 mAP) and approaching SGC‑Net (12.46 mAP).

Qualitative results illustrate that AgentHOI can discover interactions beyond dataset annotations, such as recognizing "walk the dog" or additional verbs like "straddle", "hold", and "swing lasso" in a "riding horse" image, highlighting its open‑world capability.

Conclusion and Future Work

AgentHOI demonstrates that HOI detection can be performed without dedicated training data by leveraging large‑scale multimodal pretraining and iterative reasoning. Current limitations include high inference cost, dependence on the underlying LLM’s reasoning ability, and occasional instability of generated prompts. Future directions involve lightweight inference pipelines, more reliable interaction verification mechanisms, and extending the approach to video and dynamic scenes.