让机器人通过反复推理关键线索，更智能地完成未知环境中的导航任务。
Foresight: Iterative Reasoning About Clues that Matter for Navigation
- 用视觉语言模型迭代生成并优化运动规划，结合语言目标与视觉上下文
- 实测任务成功率提升37%，每任务干预减少52%，可在实时设备运行
- 支持开放集线索发现，适合需要自主决策的机器人导航研究
从稀疏语言指令中实现开放世界无地图导航，需解决目标描述不明确的问题，并判断哪些环境线索对抵达目标重要。例如，到达视野外的目的地可能需要理解坡道、标志或绕行路径来确定方向或路线。现有方法受限于已知导航因素和封闭类别，或在运动规划前就确定线索，遗漏依赖规划的线索。我们提出Foresight，一个测试时框架，通过微调的视觉语言模型（VLM）在图像空间中交替生成运动计划并基于语言目标与视觉上下文进行批判，后续计划依据先前批判调整，实现执行前的迭代优化。为对齐开放集行为偏好，我们从人类反馈中学习奖励模型，并在计划-批判循环中使用强化学习对VLM进行后训练。在离线评估及6个真实环境中，相较于最先进的测试时推理和基础模型基线，Foresight平均任务成功率提升37%，每任务干预减少52%，且可在Jetson AGX Orin上实时运行。代码、数据与训练细节将公开，以推动测试时推理在机器人运动优化中的研究。更多视频见：https://amrl.cs.utexas.edu/foresight
原文摘要 · Abstract (English)
Open-world mapless navigation from sparse language instructions requires resolving underspecified goals and inferring which environmental cues are relevant for reaching the goal. For instance, reaching an out-of-view destination may require interpreting ramps, signs, or detours that reveal where to go or which route to take. Prior works are limited by their reliance on known navigation factors and closed-set factor categories, or identify cues before motion planning and miss plan-dependent cues. We argue that pretrained Vision-Language Models (VLMs) can discover novel instruction-relevant cues, but require adaptation to focus on which cues matter and how they should influence motion planning. We realize these ideas in Foresight, a test-time framework in which a finetuned VLM alternates between proposing image-space motion plans and critiquing them using the language goal and visual context. Subsequent plans are conditioned on prior critiques, enabling iterative motion refinement before execution. To align plan critiques and refinements with open-set behavior preferences, we learn a reward model from human feedback and use it to post-train the VLM with reinforcement learning in the plan-critique loop. In offline evaluations and 6 real-world environments, Foresight improves average task success by 37% and reduces interventions per mission by 52% relative to state-of-the-art test-time reasoning and foundation-model baselines, while running in real-time on a Jetson AGX Orin. We will release code, data, and training details to support future work on test-time reasoning for robot motion refinement. Additional videos at: https://amrl.cs.utexas.edu/foresight
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。