arXiv:2503.18950cs.CV2025-03被引 7

让视频生成精准聚焦指定物体,实现人与物的可控交互。

Target-Aware Video Diffusion Models

  • 用目标掩码和文本提示控制动作,引入特殊标记编码目标位置。
  • 在关键注意力区域施加损失,使模型生成时准确对齐目标物体。
  • 适用于零样本3D动作合成与长视频创作,提升物理合理性。

我们提出一种目标感知视频扩散模型,可根据输入图像生成视频,其中演员在执行指定动作时与特定目标发生交互。目标由分割掩码定义,动作通过文本提示描述。核心动机是将目标感知融入视频生成,使演员能对指定物体执行定向操作。这使得视频扩散模型可作为运动规划器,利用大规模视频生成模型的先验知识,生成合理的人-物交互预测。我们在基线模型基础上扩展,将目标掩码作为额外输入,并引入特殊标记,编码目标的空间信息并嵌入文本提示中。通过自建数据集微调模型,采用额外的交叉注意力损失,使该标记对应的交叉注意力图与输入目标掩码对齐。为进一步提升性能,我们仅在语义相关性最高的注意力区域和Transformer层应用此损失。实验表明,本模型在生成演员准确作用于指定目标的视频方面优于现有方法。我们进一步验证其在两个下游任务中的有效性:具备物理合理性的零样本3D人-物交互动作合成,以及长期视频内容生成。

原文摘要 · Abstract (English)

We present a target-aware video diffusion model that generates videos from an input image, in which an actor interacts with a specified target while performing a desired action. The target is defined by a segmentation mask, and the action is described through a text prompt. Our key motivation is to incorporate target awareness into video generation, enabling actors to perform directed actions on designated objects. This enables video diffusion models to act as motion planners, producing plausible predictions of human-object interactions by leveraging the priors of large-scale video generative models. We build our target-aware model by extending a baseline model to incorporate the target mask as an additional input. To enforce target awareness, we introduce a special token that encodes the target's spatial information within the text prompt. We then fine-tune the model with our curated dataset using an additional cross-attention loss that aligns the cross-attention maps associated with this token with the input target mask. To further improve performance, we selectively apply this loss to the most semantically relevant attention regions and transformer blocks. Experimental results show that our target-aware model outperforms existing solutions in generating videos where actors interact accurately with the specified targets. We further demonstrate its efficacy in two downstream applications: zero-shot 3D HOI motion synthesis with physical plausibility and long-term video content creation.

视频生成扩散模型人物交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。