用强化学习微调提升图结构导航模型,让机器人更懂自然语言指令。
ETP-R1: Evolving Topological Planning with Reinforcement Fine-tuning for Vision-Language Navigation in Continuous Environments
- 构建大规模高质量指令数据集,结合双任务联合预训练增强语言-路径映射能力。
- 首次在图结构模型中应用闭环在线强化学习微调,性能超越现有方法。
- 适合关注视觉语言导航与强化学习融合的研究者或开发者。
视觉语言导航在连续环境(VLN-CE)中要求智能体根据自然语言指令在真实空间中导航至目标。现有基于图的方法通过拓扑地图抽象环境并简化动作空间为路径点选择,虽结构清晰但难以利用大规模数据和先进训练范式,落后于基于大视觉语言模型(LVLM)的方法。本文提出ETP-R1框架,将数据规模化与强化微调(RFT)范式引入图结构模型。首先使用Gemini API构建高质量、低幻觉的大规模预训练数据集,包含多样化拓扑轨迹指令;其次联合R2R与RxR任务数据进行联合预训练。在此基础上,设计三阶段训练流程,首次实现基于组相对策略优化(GRPO)的闭环在线强化微调。大量实验表明,该方法在R2R-CE与RxR-CE基准上全面达到新最优性能。代码已开源。
原文摘要 · Abstract (English)
Vision-Language Navigation in Continuous Environments (VLN-CE) requires an embodied agent to navigate towards target in continuous environments, following natural language instructions. While current graph-based methods offer an efficient, structured approach by abstracting the environment into a topological map and simplifying the action space to waypoint selection, they lag behind methods based on Large Vision-Language Models (LVLMs) in leveraging large-scale data and advanced training paradigms. In this paper, we try to bridge this gap by introducing ETP-R1, a framework that applies the paradigm of scaling up data and Reinforcement Fine-Tuning (RFT) to a graph-based VLN-CE model. To build a strong foundation, we first construct a high-quality, large-scale pretraining dataset using the Gemini API. This dataset consists of diverse, low-hallucination instructions for topological trajectories, providing rich supervision for our graph-based policy to map language to topological paths. This foundation is further strengthened by unifying data from both R2R and RxR tasks for joint pretraining. Building on this, we introduce a three-stage training paradigm, which culminates in the first application of closed-loop, online RFT to a graph-based VLN-CE model, powered by the Group Relative Policy Optimization (GRPO) algorithm. Extensive experiments demonstrate that our approach is highly effective, establishing new state-of-the-art performance across all major metrics on both the R2R-CE and RxR-CE benchmarks. Our code is available at https://github.com/Cepillar/ETP-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。