用接触信号统一重建单目视频中手与物体的4D交互,提升真实场景下的精度与一致性。
CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction

- 以接触为显式耦合信号,统一建模手与物体的4D运动与姿态
- 在挑战性数据上实现更优的物体重建、物理合理性与时间连续性
- 适合做真实世界交互数据挖掘、场景感知生成与规划的研究者
我们探讨是否可将日常开放世界单目视频转化为可复用的4D交互基元:关节化手部运动、随时间变化的物体6D位姿与形状,以及接触发生的时空位置。这一能力可支持大规模真实交互数据挖掘,并推动场景感知的合成与规划。然而,从复杂单目视频中重建手-物体交互(HOI)仍具挑战:现有方法常依赖已知物体或受控场景,且独立估计的手与物体会因遮挡、杂乱和未知几何而错位。为此,我们提出CHOIR——一种面向单目相机的接触感知HOI重建框架,利用接触作为手与物体间的显式耦合信号。CHOIR首先基于开放世界视觉先验初始化一个粗粒度、无接触依赖的4D HOI序列;随后引入生成式空间校正模块,预测射线深度修正并校正手-物相对位置,进而获得校正几何上的初始逐帧接触对应关系;最后通过带动态更新接触约束的联合优化,强制满足几何、时间与接触一致性。在受控与挑战性视频上的实验表明,CHOIR在物体重建、物理合理性与时间一致性方面均优于当前最优方法。
原文摘要 · Abstract (English)
We ask whether everyday open-world monocular videos can be turned into reusable 4D interaction primitives: articulated hand motion, object shape with 6D pose over time, and the when/where of contact. Such a capability would enable scalable mining of real interactions and, beyond reconstruction, support scene-aware synthesis and planning. However, reconstructing hand-object interaction (HOI) from challenging monocular videos remains difficult: methods often assume known objects or curated scenes, and separately estimated hands and objects easily become misaligned under clutter, occlusion, and unseen object geometries. Targeting this setting, we present CHOIR, a Contact-aware HOI Reconstruction framework for a monocular camera, using contact as an explicit coupling signal between hands and objects. CHOIR first initializes a coarse, contact-agnostic 4D HOI sequence from open-world visual priors. It then introduces a generative HOI spatial rectification module to predict ray-depth corrections and rectify hand-object relative placement, then derive initial per-frame contact correspondences on the rectified geometry. Last, a contact-aware joint optimization with dynamically updated contact constraints enforces geometric, temporal, and contact consistency. Experiments on controlled and challenging videos show that CHOIR improves object reconstruction, physical plausibility, and temporal consistency over state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。