从单张图片和文字预测手部动作与物体接触点。
How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions
- 用向量量化模型提取手部姿态与接触点的潜在代码。
- 通过Transformer解码器预测交互轨迹，准确率优于基线方法。
- 支持跨物体、动作、场景的泛化，适合虚拟交互应用。
我们解决了一个新问题：给定单张RGB图像、动作文本和物体上的一个3D接触点，预测3D手部运动与接触图（即交互轨迹）。方法包括：(1) 交互代码本：基于VQVAE学习手部姿态与接触点的潜在代码，有效对齐交互轨迹；(2) 交互预测器：采用Transformer解码器，通过索引模块从代码本中检索潜在可用性来预测交互轨迹。为训练模型，我们构建数据引擎，从多样化的HoloAssist数据集中提取3D手部姿态与接触轨迹。在基准测试中，我们的数据集比现有工作大2.5至10倍，涵盖更丰富的物体与交互类型。实验表明，该方法在所有设置下均优于Transformer与扩散模型基线，具备强泛化能力。
原文摘要 · Abstract (English)
We tackle the novel problem of predicting 3D hand motion and contact maps (or Interaction Trajectories) given a single RGB view, action text, and a 3D contact point on the object as input. Our approach consists of (1) Interaction Codebook: a VQVAE model to learn a latent codebook of hand poses and contact points, effectively tokenizing interaction trajectories, (2) Interaction Predictor: a transformer-decoder module to predict the interaction trajectory from test time inputs by using an indexer module to retrieve a latent affordance from the learned codebook. To train our model, we develop a data engine that extracts 3D hand poses and contact trajectories from the diverse HoloAssist dataset. We evaluate our model on a benchmark that is 2.5-10X larger than existing works, in terms of diversity of objects and interactions observed, and test for generalization of the model across object categories, action categories, tasks, and scenes. Experimental results show the effectiveness of our approach over transformer & diffusion baselines across all settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。