提出高效视觉语言导航新基线,解决推理慢、训练漏动作、数据收集低效问题。
Efficient-VLN: A Simple yet Strong Baseline for Efficient Vision-Language Navigation
- 推理时复用键值缓存并连续处理位置编码,仅处理新帧实现实时响应。
- 训练中使用动作隔离掩码打包数据,提升吞吐且避免动作泄露,缩小训练与推理差距。
- 自适应DAgger动态平衡自主探索与专家指导,提升纠错能力且不增加算力负担。
尽管多模态大语言模型在视觉语言导航任务中展现出巨大潜力,现有智能体仍受制于推理、训练和数据收集中的系统性瓶颈。具体表现为:因重复处理视觉历史导致延迟过高,序列打包训练中出现动作泄露,以及自纠正数据收集阶段探索效率低下。为此,我们提出Efficient-VLN,通过三个简单而有效的机制系统性地解决这些问题:(1) 推理阶段引入连续位置编码的键值缓存复用,使模型每步仅处理新观测帧,实现实时推理;(2) 训练阶段采用带动作隔离掩码的打包训练,提升吞吐量的同时有效防止动作泄露,弥合训练-推理差距;(3) 数据收集阶段使用自适应DAgger,动态平衡自主探索与专家引导,增强错误恢复能力而不增加计算开销。大量实验表明,Efficient-VLN在R2R-CE(73.2%成功率)和RxR-CE(75.6%成功率)基准上显著超越现有水平,相较先前最优模型StreamVLN降低28%延迟,确立了基于流式大模型导航的新范式。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) have demonstrated significant promise in Vision-Language Navigation (VLN), existing agents remain heavily constrained by systemic bottlenecks across inference, training, and data collection. Specifically, they suffer from prohibitive latency due to visual history reprocessing, action leakage during sequence-packed training, and suboptimal exploration in self-correction data collection. To overcome these intertwined challenges, we present Efficient-VLN, a highly efficient and robust baseline that systematically resolves these issues through three simple-yet-effective mechanisms. (1) Inference: We introduce KV-cache reuse with contiguous RoPE, enabling the model to process only the newly observed frame at each step for real-time inference. (2) Training: We propose packed training with an action-isolating mask to accelerate throughput while effectively bridging the training-inference gap by preventing action leakage. (3) Data Collection: We employ an Adaptive DAgger to dynamically balance autonomous exploration and oracle guidance, enhancing error-recovery capability without escalating computational costs. Extensive evaluations show that Efficient-VLN significantly advances the state-of-the-art across the R2R-CE (73.2% SR) and RxR-CE (75.6% SR) benchmarks. Meanwhile, it yields a 28% latency reduction compared to the previous state-of-the-art StreamVLN, establishing a new paradigm for streaming MLLM-based navigation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。