arXiv:2512.07394cs.CV2025-12被引 2

基于手部交互时间线,实现第一视角视频中物体的精准三维重建。

Reconstructing Objects along Hand Interaction Timelines in Egocentric Video

  • 从物体视角定义手部交互时间线,建模姿态约束并传播
  • 在无3D真值下,稳定抓握重建提升6.2%-11.3%,时间线重建提升24.5%
  • 适用于日常场景第一视角视频,适合关注物体动态重建的研究者

我们提出物体沿手部交互时间线重建(ROHIT)任务。首先从刚性物体视角定义手部交互时间线(HIT):物体先相对于场景静止,随后接触并被手持,姿态发生变化,接着进入稳定抓握使用阶段,最后释放回静止状态。我们建模该时间线上的姿态约束,提出受限优化与传播(COP)框架,通过姿态传播实现更优重建。重点聚焦于稳定抓握时段——即手部持续稳定接触物体的阶段,无需3D真值即可高效标注、研究与评估。在两个第一视角数据集HOT3D和EPIC-Kitchens上验证:在HOT3D中构建1.2K个稳定抓握片段,在EPIC-Kitchens中标注2.4K个片段,涵盖390个实例、9类物体,来自141个真实环境。无3D真值时,采用2D投影误差评估重建效果。定量结果表明,COP在稳定抓握重建上提升6.2%-11.3%,在整条时间线重建上最高提升24.5%。

原文摘要 · Abstract (English)

We introduce the task of Reconstructing Objects along Hand Interaction Timelines (ROHIT). We first define the Hand Interaction Timeline (HIT) from a rigid object's perspective. In a HIT, an object is first static relative to the scene, then is held in hand following contact, where its pose changes. This is usually followed by a firm grip during use, before it is released to be static again w.r.t. to the scene. We model these pose constraints over the HIT, and propose to propagate the object's pose along the HIT enabling superior reconstruction using our proposed Constrained Optimisation and Propagation (COP) framework. Importantly, we focus on timelines with stable grasps - i.e. where the hand is stably holding an object, effectively maintaining constant contact during use. This allows us to efficiently annotate, study, and evaluate object reconstruction in videos without 3D ground truth. We evaluate our proposed task, ROHIT, over two egocentric datasets, HOT3D and in-the-wild EPIC-Kitchens. In HOT3D, we curate 1.2K clips of stable grasps. In EPIC-Kitchens, we annotate 2.4K clips of stable grasps including 390 object instances across 9 categories from videos of daily interactions in 141 environments. Without 3D ground truth, we utilise 2D projection error to assess the reconstruction. Quantitatively, COP improves stable grasp reconstruction by 6.2-11.3% and HIT reconstruction by up to 24.5% with constrained pose propagation.

三维重建第一视角手部交互姿态传播

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。