融合离线模仿与在线探索,提升视觉语言导航的纠错与泛化能力。
Joint On-and-Off Policy Learning for Vision-and-Language Navigation

- 分三阶段训练:先模仿专家轨迹,再用DAgger生成探索路径,最后联合优化策略。
- 在R2R和RxR基准上分别达到69.9%和68.0%成功率,R2R刷新最佳纪录。
- 适合关注智能体导航、多模态强化学习的研究者与开发者。
视觉-语言导航(VLN)要求智能体根据自然语言指令在物理世界中导航。近年来,基于视觉-语言模型(VLM)的方法发展迅速,主要分为两类:一是基于专家示范的模仿学习(IL),并采用数据聚合(DAgger)算法增强错误恢复能力;二是基于可验证奖励的强化学习(RL),以提升推理与探索能力。二者缺乏有效融合。本文提出JOP-VLN框架,通过三阶段训练实现离线模仿与在线探索的协同:首先在专家示范上进行模仿学习,获取基础导航技能;其次使用DAgger生成启发式探索轨迹,用于进一步模仿学习以提升错误恢复能力;最后采用联合在线与离线策略学习,结合高熵轨迹采样提升RL效率,并引入以错误纠正为导向的轨迹排序策略实现有效纠错。大量实验表明,JOP-VLN在VLN-CE R2R和RxR基准上分别取得69.9%和68.0%的成功率,刷新R2R基准的最先进水平。
原文摘要 · Abstract (English)
Vision-and-Language Navigation (VLN) necessitates an embodied agent to navigate in the physical world by adhering to natural language instructions. Recent advancements in Vision-Language Models (VLM) have propelled the development of VLM-based VLN methods with two predominant paradigms: (1) imitation learning (IL) on expert demonstrations, followed by the Dataset Aggregation (DAgger) algorithm to bolster error recovery capabilities; (2) reinforcement learning (RL) driven by verifiable rewards to enhance reasoning and exploration. A notable gap is the absence of integration between these two distinct paradigms. This paper introduces JOP-VLN, a novel VLN framework that synergistically combines off-policy imitation learning and on-policy exploration within a three-stage training pipeline. Initially, IL is employed on expert demonstrations to acquire basic navigation skills. Subsequently, the DAgger algorithm is utilized to generate heuristic exploration trajectories, which are then used for imitation learning to improve error recovery capabilities. Finally, a joint on-and-off policy learning framework is implemented, featuring high-entropy trajectory sampling to enhance RL training efficiency and an error-correction-prioritized trajectory sorting strategy for effective error correction. Extensive experiments demonstrate the efficacy of JOP-VLN, achieving success rates of 69.9% and 68.0% on the VLN-CE R2R and RxR benchmarks, respectively, setting a new state-of-the-art on R2R. Project page: https://qingrongh.github.io/JOP-VLN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。