用分层记忆系统解决开源导航模型遗忘问题，性能接近闭源模型
HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory System
- 引入分层记忆系统增强视觉回忆与长期定位能力
- 在仿真和真实环境均实现双倍于当前开源最佳方法的性能
- 适合关注开源大模型可靠性的研究者与开发者
基于大语言模型的智能体在视觉语言导航任务中展现出出色的零样本表现。然而，多数零样本方法依赖闭源大模型作为导航器，面临高令牌成本和潜在数据泄露风险。近期工作尝试使用开源大模型结合时空思维链框架，但仍显著落后于闭源模型。本文通过深入分析导航过程，识别出关键问题——导航遗忘（Navigation Amnesia），该问题导致导航失败并加剧开源与闭源方法间的差距。为此，我们提出HiMemVLN，将分层记忆系统融入多模态大模型，提升视觉感知回忆与长期定位能力，缓解遗忘问题，显著改善智能体导航性能。在仿真与真实世界环境中的大量实验表明，HiMemVLN性能近乎达到当前开源最优方法的两倍。代码已公开于https://github.com/lvkailin0118/HiMemVLN。
原文摘要 · Abstract (English)
LLM-based agents have demonstrated impressive zero-shot performance in vision-language navigation (VLN) tasks. However, most zero-shot methods primarily rely on closed-source LLMs as navigators, which face challenges related to high token costs and potential data leakage risks. Recent efforts have attempted to address this by using open-source LLMs combined with a spatiotemporal CoT framework, but they still fall far short compared to closed-source models. In this work, we identify a critical issue, Navigation Amnesia, through a detailed analysis of the navigation process. This issue leads to navigation failures and amplifies the gap between open-source and closed-source methods. To address this, we propose HiMemVLN, which incorporates a Hierarchical Memory System into a multimodal large model to enhance visual perception recall and long-term localization, mitigating the amnesia issue and improving the agent's navigation performance. Extensive experiments in both simulated and real-world environments demonstrate that HiMemVLN achieves nearly twice the performance of the open-source state-of-the-art method. The code is available at https://github.com/lvkailin0118/HiMemVLN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。