arXiv:2510.25268cs.ROcs.AI2025-10被引 1

用语言指令生成物体关节动作的双手操作序列,支持真实物理交互。

SynHLMA:Synthesizing Hand Language Manipulation for Articulated Object with Discrete Human Object Interaction Representation

  • 用离散交互表示建模每帧手物关系,结合语言嵌入统一建模空间。
  • 在自建HAOI-lang数据集上优于现有方法,可完成抓取、预测与插值任务。
  • 适用于机器人灵巧操作生成,支持从模仿学习中执行复杂抓握。

根据语言指令生成手部对可动物体的操作序列是具身智能与虚拟/增强现实应用的重要课题。当涉及手部关节物体交互(HAOI)时,不仅需满足物体功能,还需生成随物体形变变化的长期操作序列。本文提出新型框架SynHLMA,用于合成手部语言操控序列。给定可动物体完整点云,采用离散的HAOI表示建模每帧手物交互;结合自然语言嵌入,通过一个HAOI操作语言模型在共享表示空间中对齐抓取过程与语言描述。引入关节感知损失,确保手部抓握跟随可动物体关节的动态变化。该方法实现三类典型任务:HAOI生成、预测与插值。在自建HAOI-lang数据集上的实验表明,相比当前最优方法,其手部操作序列生成性能显著提升。此外,我们展示了机器人抓握应用,通过模仿学习利用SynHLMA提供的操作序列实现灵巧抓握。代码与数据集将公开。

原文摘要 · Abstract (English)

Generating hand grasps with language instructions is a widely studied topic that benefits from embodied AI and VR/AR applications. While transferring into hand articulatied object interaction (HAOI), the hand grasps synthesis requires not only object functionality but also long-term manipulation sequence along the object deformation. This paper proposes a novel HAOI sequence generation framework SynHLMA, to synthesize hand language manipulation for articulated objects. Given a complete point cloud of an articulated object, we utilize a discrete HAOI representation to model each hand object interaction frame. Along with the natural language embeddings, the representations are trained by an HAOI manipulation language model to align the grasping process with its language description in a shared representation space. A joint-aware loss is employed to ensure hand grasps follow the dynamic variations of articulated object joints. In this way, our SynHLMA achieves three typical hand manipulation tasks for articulated objects of HAOI generation, HAOI prediction and HAOI interpolation. We evaluate SynHLMA on our built HAOI-lang dataset and experimental results demonstrate the superior hand grasp sequence generation performance comparing with state-of-the-art. We also show a robotics grasp application that enables dexterous grasps execution from imitation learning using the manipulation sequence provided by our SynHLMA. Our codes and datasets will be made publicly available.

手部操作语言控制机器人抓握可动物体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。