用双隐式记忆分离语义与空间信息,提升视觉语言导航效率
JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation

- 采用双隐式神经记忆,分别建模语义与空间信息
- 成功率达92.3%,比多模态输入方法提升35.5个百分点
- 适合研究智能体导航与具身认知的学者参考
视觉-语言导航要求智能体在未见过的环境中,根据自然语言指令和连续视频流进行导航。当前方法依赖显式语义记忆(如构建文本认知地图或存储历史视觉帧),存在空间信息丢失、计算冗余和内存膨胀问题。受人类导航中左脑语义理解与右脑空间认知的启发,我们提出JanusVLN框架,采用双隐式神经记忆,将空间几何与视觉语义信息作为独立、紧凑、固定大小的神经表示。该框架首先引入三维先验知识增强仅基于RGB输入模型的空间推理能力;随后,由空间几何与视觉语义编码器的历史键值缓存构成双隐式记忆,仅保留初始与滑动窗口内的键值,避免冗余计算,实现高效增量更新。大量实验表明,JanusVLN超越20余种近期方法,达到新SOTA性能:相较于使用多数据类型输入的方法,成功率提升10.5–35.5个百分点;相比使用更多RGB训练数据的方法,提升3.6–10.8个百分点。这证明双隐式神经记忆是一种有前景的新范式,为未来视觉语言导航研究提供方向。
原文摘要 · Abstract (English)
Vision-and-Language Navigation requires an embodied agent to navigate through unseen environments, guided by natural language instructions and a continuous video stream. Recent advances in VLN have been driven by the powerful semantic understanding of Multimodal Large Language Models. However, these methods typically rely on explicit semantic memory, such as building textual cognitive maps or storing historical visual frames. This type of method suffers from spatial information loss, computational redundancy, and memory bloat, which impede efficient navigation. Inspired by the implicit scene representation in human navigation, analogous to the left brain's semantic understanding and the right brain's spatial cognition, we propose JanusVLN, a novel VLN framework featuring a dual implicit neural memory that models spatial-geometric and visual-semantic memory as separate, compact, and fixed-size neural representations. This framework first extends the MLLM to incorporate 3D prior knowledge from the spatial-geometric encoder, thereby enhancing the spatial reasoning capabilities of models based solely on RGB input. Then, the historical key-value caches from the spatial-geometric and visual-semantic encoders are constructed into a dual implicit memory. By retaining only the KVs of tokens in the initial and sliding window, redundant computation is avoided, enabling efficient incremental updates. Extensive experiments demonstrate that JanusVLN outperforms over 20 recent methods to achieve SOTA performance. For example, the success rate improves by 10.5-35.5 compared to methods using multiple data types as input and by 3.6-10.8 compared to methods using more RGB training data. This indicates that the proposed dual implicit neural memory, as a novel paradigm, explores promising new directions for future VLN research. Ours project page: https://miv-xjtu.github.io/JanusVLN.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。