arXiv:2505.20710cs.CV2025-05被引 4

用空间目标连接指令与动作,让机器人更懂人话地追踪目标

Hierarchical Instruction-aware Embodied Visual Tracking

  • 将人类指令转为具体空间位置,再由策略执行
  • 在10个环境上实现跨场景稳定追踪,成功率超90%
  • 适合需要理解复杂指令的机器人视觉追踪任务

用户中心的具身视觉追踪(UC-EVT)因高层指令与底层动作间存在巨大鸿沟,对基于强化学习的模型构成新挑战。尽管语言模型(如LLMs、VLMs、VLAs)提升了指令理解能力,但普遍存在推理速度慢(LLMs、VLMs)或泛化能力差(VLAs)的问题。为此,我们提出层级指令感知具身视觉追踪(HIEVT)代理,通过空间目标作为中介,连接指令理解与动作生成。HIEVT首先引入基于LLM的语义-空间目标对齐器,将多样化的用户指令转化为直接标注目标空间位置的空间目标;随后,基于强化学习的自适应目标对齐策略(通用离线策略)使追踪器能按空间目标精准定位目标。为评估任务性能,我们收集了超过一千万条轨迹用于训练,并在1个已见环境和9个未见复杂环境中进行评测。大量实验与真实部署表明,HIEVT在不同环境、目标动态变化及复杂指令组合下均表现出强鲁棒性与泛化能力。完整项目见 https://sites.google.com/view/hievt。

原文摘要 · Abstract (English)

User-Centric Embodied Visual Tracking (UC-EVT) presents a novel challenge for reinforcement learning-based models due to the substantial gap between high-level user instructions and low-level agent actions. While recent advancements in language models (e.g., LLMs, VLMs, VLAs) have improved instruction comprehension, these models face critical limitations in either inference speed (LLMs, VLMs) or generalizability (VLAs) for UC-EVT tasks. To address these challenges, we propose \textbf{Hierarchical Instruction-aware Embodied Visual Tracking (HIEVT)} agent, which bridges instruction comprehension and action generation using \textit{spatial goals} as intermediaries. HIEVT first introduces \textit{LLM-based Semantic-Spatial Goal Aligner} to translate diverse human instructions into spatial goals that directly annotate the desired spatial position. Then the \textit{RL-based Adaptive Goal-Aligned Policy}, a general offline policy, enables the tracker to position the target as specified by the spatial goal. To benchmark UC-EVT tasks, we collect over ten million trajectories for training and evaluate across one seen environment and nine unseen challenging environments. Extensive experiments and real-world deployments demonstrate the robustness and generalizability of HIEVT across diverse environments, varying target dynamics, and complex instruction combinations. The complete project is available at https://sites.google.com/view/hievt.

具身智能视觉追踪指令理解强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。