用检索增强框架提升大模型导航效率,不改模型也能显著提效。
Learning to Retrieve Navigable Candidates for Efficient Vision-and-Language Navigation
- 在任务和步骤两级引入轻量检索,分别提供相似路径示例和筛选候选方向
- 在R2R数据集上成功率达63.1%,未见环境也稳定提升,且推理更高效
- 无需微调大模型,模块化设计适合快速部署到各类视觉语言导航系统
视觉-语言导航(VLN)要求智能体根据自然语言指令在未见过的环境中导航。近年方法越来越多地使用大语言模型(LLM)作为高层导航器,因其具备灵活性和推理能力。然而,基于提示的LLM导航常因重复解析指令、在冗长嘈杂的可行动作中推理而效率低下。本文提出一种检索增强框架,在不修改或微调底层语言模型的前提下,提升LLM-based VLN的效率与稳定性。方法在两个互补层级引入检索:在任务层面,通过指令嵌入检索器选取语义相近的成功导航轨迹作为上下文示例,为指令理解提供任务特定先验;在步骤层面,通过模仿学习的候选检索器在LLM推理前剔除无关方向,降低动作歧义与提示复杂度。两个检索模块均轻量、模块化,且独立训练。在Room-to-Room(R2R)基准上评估,实验结果表明在已见与未见环境中,成功率达63.1%、最优成功率达75.2%、SPL达60.3%,均有持续提升。消融实验显示,指令级示例检索与候选剪枝分别贡献全局引导与逐步决策效率的互补优势。结果表明,检索增强决策支持是提升基于大模型的视觉语言导航的有效且可扩展策略。
原文摘要 · Abstract (English)
Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions and navigate through previously unseen environments. Recent approaches increasingly employ large language models (LLMs) as high-level navigators due to their flexibility and reasoning capability. However, prompt-based LLM navigation often suffers from inefficient decision-making, as the model must repeatedly interpret instructions from scratch and reason over noisy and verbose navigable candidates at each step. In this paper, we propose a retrieval-augmented framework to improve the efficiency and stability of LLM-based VLN without modifying or fine-tuning the underlying language model. Our approach introduces retrieval at two complementary levels. At the episode level, an instruction-level embedding retriever selects semantically similar successful navigation trajectories as in-context exemplars, providing task-specific priors for instruction grounding. At the step level, an imitation-learned candidate retriever prunes irrelevant navigable directions before LLM inference, reducing action ambiguity and prompt complexity. Both retrieval modules are lightweight, modular, and trained independently of the LLM. We evaluate our method on the Room-to-Room (R2R) benchmark. Experimental results demonstrate consistent improvements in Success Rate, Oracle Success Rate, and SPL on both seen and unseen environments. Ablation studies further show that instruction-level exemplar retrieval and candidate pruning contribute complementary benefits to global guidance and step-wise decision efficiency. These results indicate that retrieval-augmented decision support is an effective and scalable strategy for enhancing LLM-based vision-and-language navigation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。