让机器人通过视觉提示交互式推理,提升复杂任务的鲁棒性。
Guide, Think, Act: Interactive Embodied Reasoning in Vision-Language-Action Models

- 用户可用点、框、轨迹等视觉提示引导机器人决策
- 在仿真环境中达成81.2%任务成功率,显著优于现有方法
- 支持故障恢复与视觉模糊场景下的交互纠错
本文提出GTA-VLA(Guide, Think, Act)框架,一种支持空间可引导的具身视觉-语言-动作模型。传统VLA模型直接从多模态输入映射到动作,虽在训练分布内有效,但在域外(OOD)场景下易失效且难以修正错误。现有具身思维链(CoT)方法虽暴露中间推理过程,却缺乏对人类空间引导的整合能力,难以解决视觉歧义或恢复失误。为此,本框架允许用户以显式视觉线索(如可操作点、框、轨迹)引导策略,后续推理过程可直接基于这些输入进行条件化。模型生成统一的空间-视觉思维链,融合外部引导与内部任务规划,实现人类视觉意图与自主决策的对齐。为便于部署,推理模块还耦合轻量级反应式动作头,实现高效执行。大量实验表明:在同域的SimplerEnv WidowX基准上,本方法达到81.2%的成功率;在域外视觉变化和空间歧义场景下,一次视觉交互即可显著提升成功率,凸显交互式推理在具身控制中失败恢复的价值。
原文摘要 · Abstract (English)
In this paper, we propose GTA-VLA(Guide, Think, Act), an interactive Vision-Language-Action (VLA) framework that enables spatially steerable embodied reasoning by allowing users to guide robot policies with explicit visual cues. Existing VLA models learn a direct "Sense-to-Act" mapping from multimodal observations to robot actions. While effective within the training distribution, such tightly coupled policies are brittle under out-of-domain (OOD) shifts and difficult to correct when failures occur. Although recent embodied Chain-of-Thought (CoT) approaches expose intermediate reasoning, they still lack a mechanism for incorporating human spatial guidance, limiting their ability to resolve visual ambiguities or recover from mistakes. To address this gap, our framework allows users to optionally guide the policy with spatial priors, such as affordance points, boxes, and traces, which the subsequent reasoning process can directly condition on. Based on these inputs, the model generates a unified spatial-visual Chain-of-Thought that integrates external guidance with internal task planning, aligning human visual intent with autonomous decision-making. For practical deployment, we further couple the reasoning module with a lightweight reactive action head for efficient action execution. Extensive experiments demonstrate the effectiveness of our approach. On the in-domain SimplerEnv WidowX benchmark, our framework achieves a state-of-the-art 81.2% success rate. Under OOD visual shifts and spatial ambiguities, a single visual interaction substantially improves task success over existing methods, highlighting the value of interactive reasoning for failure recovery in embodied control. More details of the project can be found here: https://github.com/FutianLabs/GTA-VLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。