用图神经网络建模双手工具操作,提升精度与泛化能力
PhysGraph: Physically-Grounded Graph-Transformer Policies for Bimanual Dexterous Hand-Tool-Object Manipulation
- 将双手工具系统构建成骨架图,每段独立编码保留局部状态
- 引入物理先验增强注意力机制,任务成功率显著提升
- 仅需51%参数量,可零样本迁移至新工具和机器人手
双臂灵巧操作工具仍面临高维状态空间和复杂接触动力学的挑战。现有方法将系统状态简化为单一向量,忽略机械手固有的结构与拓扑信息。本文提出PhysGraph,一种专为复杂双臂手-工具-物体操作设计的物理引导图变压器策略。不同于以往工作,我们以运动学图为形式表示双臂系统,并采用逐段节点化方式保留细粒度局部状态信息。提出物理引导偏置生成器,直接在注意力机制中注入运动学空间距离、动态接触状态、几何邻近性及解剖特性等结构先验。使策略能显式推理物理交互,而非依赖稀疏奖励隐式学习。大量实验表明,PhysGraph在操作精度和任务成功率上显著优于基线模型ManipTrans,且参数量仅为后者的51%。此外,其固有的拓扑灵活性实现对未见工具/物体几何形状的定性零样本迁移,并可在三种机器人手(Shadow、Allegro、Inspire)上训练。
原文摘要 · Abstract (English)
Bimanual dexterous manipulation for tool use remains a formidable challenge in robotics due to the high-dimensional state space and complicated contact dynamics. Existing methods naively represent the entire system state as a single configuration vector, disregarding the rich structural and topological information inherent to articulated hands. We present PhysGraph, a physically-grounded graph transformer policy designed explicitly for challenging bimanual hand-tool-object manipulation. Unlike prior works, we represent the bimanual system as a kinematic graph and introduce per-link tokenization to preserve fine-grained local state information. We propose a physically-grounded bias generator that injects structural priors directly into the attention mechanism, including kinematic spatial distance, dynamic contact states, geometric proximity, and anatomical properties. This allows the policy to explicitly reason about physical interactions rather than learning them implicitly from sparse rewards. Extensive experiments show that PhysGraph significantly outperforms baseline - ManipTrans in manipulation precision and task success rates while using only 51% of the parameters of ManipTrans. Furthermore, the inherent topological flexibility of our architecture shows qualitative zero-shot transfer to unseen tool/object geometries, and is sufficiently general to be trained on three robotic hands (Shadow, Allegro, Inspire).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。