arXiv:2608.22449cs.ROcs.AI2026-08

将操作规划作为中间表示,提升第一视角手部动作预测精度

EMPIRE: Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecasting

论文配图:EMPIRE: Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecasting
图 1 · 摘自论文原文
  • 分两阶段建模:先学操作规划,再生成手部动作
  • 在相同条件下达到84.53毫米的MPJPE和38.97毫米指间误差
  • 适合需要精准手物交互预测的机器人与虚拟现实应用

从第一视角观测中预测灵巧手部运动是智能交互系统的基础。现有基于视觉语言模型的方法通常直接将观测映射为未来动作,忽略了支配手物交互的潜在操作过程。此外,端到端优化使操作学习与动作生成耦合,导致动作生成梯度干扰预训练的操作感知表征。为此,我们提出EMPIRE,一种两阶段框架,引入显式操作规划作为第一视角手部动作预测的中间表示。第一阶段:学习规划。EMPIRE从多模态上下文中学习显式操作规划,捕捉手物交互的演进过程。第二阶段:学习执行。一个动作生成器基于冻结的规划表征合成未来双手动作,防止动作生成梯度影响操作规划。为支持该方法,我们进一步构建了包含650,910个训练窗口的双人手部动作预测数据集EMPIRE-651K,涵盖111项任务,每项任务配有显式的单手操作规划。在相同的训练与评估协议下,EMPIRE达到最先进的预测精度,MPJPE为84.53毫米,指间相对误差为38.97毫米。代码与数据集已公开于https://github.com/wangwen-banban/EMPIRE。

原文摘要 · Abstract (English)

Forecasting dexterous hand motions from egocentric observations is fundamental to intelligent interactive systems. Existing VLM-based methods typically map observations directly to future motions, overlooking the underlying manipulation process that governs hand-object interactions. Moreover, end-to-end optimization couples manipulation learning with motion synthesis, causing motion-generation gradients to interfere with the pre-learned manipulation-aware representations. To overcome these limitations, we propose EMPIRE, a two-stage framework that introduces Explicit Manipulation Planning as an Intermediate Representation for Egocentric hand-motion forecasting. Stage I: Learn to Plan. EMPIRE first learns explicit manipulation plans from multimodal context to capture the progression of hand-object interactions. Stage II: Learn to Act. A motion generator synthesizes future bimanual hand motions conditioned on frozen planner representations, preventing motion-generation gradients from affecting manipulation planning. To support our method, we further construct EMPIRE-651K, a bimanual hand-motion forecasting dataset comprising 650,910 training windows across 111 tasks, each paired with an explicit per-hand manipulation plan. Under identical training and evaluation protocols, EMPIRE achieves state-of-the-art forecasting accuracy, with an MPJPE of 84.53 mm and a finger-relative error of 38.97mm. We release the code and dataset at https://github.com/wangwen-banban/EMPIRE.

动作预测手部建模交互理解中间表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。