arXiv:2606.23675cs.CV2026-06

用参考图精准控制人物与物体交互的细节动作。

IMAGIN-4D: Image-Guided Controllable Interaction Generation

论文配图:IMAGIN-4D: Image-Guided Controllable Interaction Generation
图 1 · 摘自论文原文
  • 将参考图像分解为时空条件,分别指导身体姿态和物体位置。
  • 生成动作与参考图匹配度更高,且能准确遵循路径点。
  • 适合需要精细交互控制的动画、机器人和虚拟现实应用。

生成人-物交互(HOI)是角色动画、机器人、AR/VR和具身AI的核心。现有方法基于文本、物体几何和稀疏路径点生成动作,但这些信号难以明确交互细节:相同提示与轨迹可产生不同抓握方式、接近方向、身体姿态、物体姿态、接触关系和布局。为此,本文引入IMAGIN-4D,一种基于扩散模型的HOI生成器,通过时空分解参考图像实现精确控制。空间上,提取受监督的交互状态标记,涵盖身体姿态、物体姿态、身体-物体接触及空间关系;时间上,对每一帧查询图像块,生成帧感知标记,使序列段可关注图像中不同视觉线索。为平衡多源信息,采用角色感知条件机制:文本、路径点与交互状态标记使用独立的AdaLN流,帧感知视觉标记则与运动标记跨注意力。由于缺乏配对图像的数据集,构建了从FullBodyManipulation(FBM)合成运动到图像的渲染管道,并设计图像一致性度量评估生成动作是否匹配参考快照。在FBM与BEHAVE数据集上的实验表明,相比单标记和统一图像条件基线,IMAGIN-4D在细粒度交互控制上表现更优,同时保持路径跟随与动作质量。代码与模型将在https://imagin4d.github.io发布。

原文摘要 · Abstract (English)

Generating human-object interactions (HOI) is central to character animation, robotics, AR/VR, and embodied AI. Recent HOI generation methods synthesize motion from text, object geometry, and sparse waypoints, controlling action semantics and object trajectories. However, these signals underspecify interaction: the same prompt and trajectory can produce different grasps, approach directions, body poses, object poses, contacts, and body-object layouts. We address this ambiguity with a reference image as a visual specification of the desired interaction snapshot. However, a single global image representation conflates distinct cues and conditions all frames on identical visual evidence. We therefore introduce IMAGIN-4D, a diffusion-based HOI generator that decomposes image conditioning spatio-temporally. For spatial conditioning, IMAGIN-4D extracts supervised interaction-state tokens for body pose, object pose, body-object contact, and spatial relationships at the depicted frame. For temporal conditioning, it computes frame-aware tokens by querying image patches per generated frame, allowing sequence segments to attend to different visual cues from the same image. To balance image, text, and waypoint cues, IMAGIN-4D uses role-aware conditioning: text, waypoints, and interaction-state tokens use separate AdaLN streams, while frame-aware visual tokens cross-attend with motion tokens. Since HOI motion datasets lack paired images, we build a synthetic motion-to-image rendering pipeline from FullBodyManipulation (FBM) and introduce an image-adherence metric to evaluate whether generated motions match the reference snapshot. Experiments on FBM and BEHAVE show that IMAGIN-4D improves fine-grained interaction control over single-token and uniformly image-conditioned baselines while preserving waypoint-following and motion quality. Code and models will be released at https://imagin4d.github.io.

交互生成扩散模型图像控制动作合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。