arXiv:2509.12618cs.ROcs.AI2025-09被引 15

让导航智能体通过多轮强化学习主动探索环境,提升路径发现能力。

ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation

  • 采用多轮强化学习实现动态环境交互与主动探索。
  • 在少量专家轨迹基础上,性能超越现有模仿学习与强化学习方法。
  • 适合研究视觉语言导航中高效探索与自主决策的学者。

视觉-语言导航(VLN)任务要求智能体根据自然语言指令在复杂环境中导航。现有基于多模态大模型的VLN方法主要依赖模仿学习(IL),常使用DAgger进行后训练以缓解协变量偏移,但成本高昂。强化学习(RL)虽具潜力,但以往方法缺乏与环境的动态互动,依赖专家轨迹进行奖励设计,限制了对多样化、合理路径的发现。为此,我们提出ActiveVLN框架,通过多轮强化学习显式支持主动探索。第一阶段用少量专家轨迹进行模仿学习以启动智能体;第二阶段,智能体迭代预测并执行动作,自动收集多样轨迹,并通过GRPO目标优化多个回溯。为提升效率,引入动态早停策略剔除长尾或可能失败的轨迹,辅以工程优化。实验表明,ActiveVLN相较模仿学习基线取得最大性能提升,优于基于DAgger和先前强化学习的后训练方法,且在使用更小模型的情况下达到与顶尖方法相当的性能。代码与数据将很快公开。

原文摘要 · Abstract (English)

The Vision-and-Language Navigation (VLN) task requires an agent to follow natural language instructions and navigate through complex environments. Existing MLLM-based VLN methods primarily rely on imitation learning (IL) and often use DAgger for post-training to mitigate covariate shift. While effective, these approaches incur substantial data collection and training costs. Reinforcement learning (RL) offers a promising alternative. However, prior VLN RL methods lack dynamic interaction with the environment and depend on expert trajectories for reward shaping, rather than engaging in open-ended active exploration. This restricts the agent's ability to discover diverse and plausible navigation routes. To address these limitations, we propose ActiveVLN, a VLN framework that explicitly enables active exploration through multi-turn RL. In the first stage, a small fraction of expert trajectories is used for IL to bootstrap the agent. In the second stage, the agent iteratively predicts and executes actions, automatically collects diverse trajectories, and optimizes multiple rollouts via the GRPO objective. To further improve RL efficiency, we introduce a dynamic early-stopping strategy to prune long-tail or likely failed trajectories, along with additional engineering optimizations. Experiments show that ActiveVLN achieves the largest performance gains over IL baselines compared to both DAgger-based and prior RL-based post-training methods, while reaching competitive performance with state-of-the-art approaches despite using a smaller model. Code and data will be released soon.

视觉语言导航强化学习主动探索多轮交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。