让手物交互更真实一致,解决视频生成中物体形象不稳的问题
GenHOI: Towards Object-Consistent Hand-Object Interaction with Temporally Balanced and Spatially Selective Object Injection
- 通过时空可控的参考物注入,保持物体跨帧一致性
- 在野外场景下相比现有方法,交互真实度提升显著
- 适合需要精确手物交互的数字人视频生成任务
手物交互(HOI)是数字人视频合成的核心挑战,要求生成物理上合理且跨帧保持物体身份的交互。尽管近期重演方法取得进展,但多在域内训练和评估,难以泛化到复杂真实场景。而全功能视频编辑模型虽更具鲁棒性,仍存在物体外观不一致问题。本文提出GenHOI,一种轻量级增强方案,对预训练视频生成模型进行参考物信息的时序均衡与空间选择性注入。为实现时序均衡,提出头滑动旋转位置编码(Head-Sliding RoPE),为不同注意力头分配时序偏移,均匀分布参考信息影响,缓解3D RoPE的时间衰减,提升长序列物体一致性。为实现空间选择性,设计两级空间注意力门,聚焦于手物交互区域并自适应调节强度,在保留背景真实感的同时增强交互准确性。在未见过的真实场景上进行大量定性和定量评估表明,GenHOI显著优于当前最先进的HOI重演与全功能视频编辑方法。
原文摘要 · Abstract (English)
Hand-Object Interaction (HOI) remains a core challenge in digital human video synthesis, where models must generate physically plausible contact and preserve object identity across frames. Although recent HOI reenactment approaches have achieved progress, they are typically trained and evaluated in-domain and fail to generalize to complex, in-the-wild scenarios. In contrast, all-in-one video editing models exhibit broader robustness but still struggle with HOI-specific issues such as inconsistent object appearance. In this paper, we present GenHOI, a lightweight augmentation to pretrained video generation models that injects reference-object information in a temporally balanced and spatially selective manner. For temporal balancing, we propose Head-Sliding RoPE, which assigns head-specific temporal offsets to reference tokens, distributing their influence evenly across frames and mitigating the temporal decay of 3D RoPE to improve long-range object consistency. For spatial selectivity, we design a two-level spatial attention gate that concentrates object-conditioned attention on HOI regions and adaptively scales its strength, preserving background realism while enhancing interaction fidelity. Extensive qualitative and quantitative evaluations on unseen, in-the-wild scenes demonstrate that GenHOI significantly outperforms state-of-the-art HOI reenactment and all-in-one video editing methods. Project page: https://xuanhuang0.github.io/GenHOI/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。