Introducing HOI-Edit: A New Benchmark and Self‑Correcting Framework for Interactive Image Editing

This article reports a newly accepted ICML 2026 paper titled Taming I2V models for Image HOI Editing: A Cognitive Benchmark and Agentic Self‑Correcting Framework , which tackles the challenging problem of editing images that involve complex human‑object interactions (HOI).

HOI‑Edit Hierarchical Cognitive Benchmark

HOI‑Edit defines three cognitive levels for HOI image editing:

L1 Basic Interaction Editing : create, remove, or modify interactions while preserving the identities of persons and objects.

L2 Contextual Spatial Understanding : correctly resolve spatial relations, e.g., selecting the right object among similar ones or placing actions in the proper location.

L3 Causal and Physical Reasoning : infer cause‑effect chains and ensure physically plausible outcomes, such as performing prerequisite actions before the final result.

The dataset contains 357 L1, 202 L2, and 146 L3 samples, covering a wide range of interaction verbs and scenarios.

HOI‑Eval Automatic Evaluation Protocol

HOI‑Eval addresses the limitation of global similarity metrics (e.g., CLIPScore) by grounding evaluation on paired image regions. The protocol consists of three steps:

Target Region Association : match person, object, and auxiliary regions between the original and edited images.

Identity Consistency Verification : check that the identities of persons and objects remain unchanged.

Interaction & Reasonableness Q&A : ask targeted questions about whether the interaction occurred, whether the action was correctly performed, whether spatial placement is accurate, and whether the process is physically reasonable.

This region‑based approach yields higher correlation with human judgments than DINOv2 or CLIP metrics (see Table 2).

SCPE Self‑Correcting Process Editing Framework

SCPE leverages the sequential video output of image‑to‑video (I2V) models to diagnose and repair failures. It comprises four agents:

Generator : produces refined video prompts from the input image, instruction, and a “playbook” of accumulated experience.

Analyzer : samples video frames and identifies failure causes such as trajectory deviation, missing steps, or implausible physics.

Reflector : abstracts a single failure case into a generalizable rule.

Curator : writes the new rule back into the playbook.

During the next generation round, the Generator consults the updated playbook, explicitly emphasizing target objects, action trajectories, preconditions, and physical consequences, thereby reducing repeat errors.

Experimental Validation

SCPE was evaluated on the HOI‑Edit benchmark using open‑source image‑editing models, commercial black‑box models, and I2V video generators. Four metrics were measured: Interaction accuracy (I), Human identity preservation (H), Object identity preservation (O), and Interaction + Q&A success (I+Q&A).

Key findings include:

Across all three cognitive levels, SCPE improves interaction scores by ~22 % (L1), ~26 % (L2), and ~22 % (L3) compared with the baseline Wan 2.2 I2V model.

Table 1 shows that SCPE outperforms strong closed‑source baselines on most interaction metrics.

Ablation (Table 3) reveals that the full SCPE pipeline (Generator + Analyzer + Reflector + Curator) achieves 0.8199 / 0.8954 on interaction and identity scores, surpassing both a simple Prompt Enhancer (0.7028 / 0.7385) and a version without the playbook (0.7625 / 0.8786).

Generalization tests with TurboDiffusion, independent evaluators, and alternative playbooks (e.g., Qwen) confirm that performance gains are robust to model and evaluator changes.

Conclusion

The work delivers a complete solution for complex HOI image editing: a hierarchical benchmark (HOI‑Edit), a fine‑grained automatic metric (HOI‑Eval), and a self‑correcting framework (SCPE) that turns video‑generated process feedback into reusable prompt improvements. Experiments demonstrate that SCPE consistently enhances interaction correctness, identity preservation, spatial understanding, and physical plausibility, establishing a new paradigm for interactive visual generation.