让机器人在移动中精准抓取,靠的是只在可靠位置注入几何信息。
GeoHAT: Geometry-Adaptive Hybrid Action Transformer for Mobile Manipulation

- 用傅里叶编码将像素级3D坐标转为几何令牌,轻量且无需额外3D模型
- 在ManiSkill-HAB上达79.3%成功率,比最强基线高23.7%
- 适合需要移动与操作协同的机器人系统研发者
全身体移动操作需协调移动底盘与机械臂在动态视角下的行为,现有方法或依赖2D特征或稀疏3D表示,缺乏稠密空间结构,且常将臂与底盘动作编码于单一向量,忽略其控制差异。同时,现有稠密融合策略在噪声深度下易破坏预训练表征,并带来高计算开销。我们提出GeoHAT,一种基于扩散模型的端到端框架,核心思想是:几何信息仅在可靠位置注入,且仅在需要时关注。该方法采用轻量级傅里叶空间编码器,将稠密像素级3D坐标映射为几何令牌,无需额外3D视觉骨干网络。这些令牌通过按令牌门控融合机制,由深度有效性调控,注入视觉基础模型特征,保留语义先验的同时增强空间理解。动作生成方面,混合式全身动作解码器将臂与底盘分解至独立子空间,各模态通过稀疏交叉注意力聚焦任务相关视觉上下文,因果时序建模捕捉同帧内协调与跨帧依赖。在ManiSkill-HAB仿真基准上的实验表明,GeoHAT实现79.3%平均成功率达,超越最强基线23.7%。真实世界多样任务实验也持续验证了其对所有基线的优越性。
原文摘要 · Abstract (English)
Whole-body mobile manipulation requires coordinating mobile base and manipulator under shifting viewpoints, posing challenges in geometric perception and action generation. Current policies either rely on 2D features or sparse 3D representations that lack dense spatial structure, and typically encode arm and base within one action vector that ignores their distinct control demands. Moreover, existing dense fusion strategies risk corrupting pretrained representations under noisy depth while incurring heavy computational overhead. We present GeoHAT, an end-to-end diffusion-based framework built on a simple principle: geometry should be injected only where reliable and attended to only where needed. GeoHAT employs a lightweight Fourier spatial encoder that maps dense per-pixel 3D coordinates into geometric tokens without an additional 3D vision backbone. These tokens are then selectively injected into vision foundation model features through per-token gated fusion modulated by depth validity, preserving the semantic prior while enriching spatial understanding. For action generation, a Hybrid Whole-Body Action Decoder decomposes arm and base into distinct subspaces and lets each action modality attend to its task-relevant visual context through sparse cross-attention, while causal temporal modeling captures intra-timestep coordination and inter-timestep dependencies. Experiments on the ManiSkill-HAB simulation benchmark demonstrate that GeoHAT achieves a 79.3% mean success rate, surpassing the strongest baseline by 23.7%. Furthermore, real-world experiments on diverse tasks also confirm consistent improvements over all baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。