提出慢快双流模型，实现低延迟的视觉语言导航。
StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling
- 采用慢快双流结构，分别处理短期对话与长期记忆。
- 在VLN-CE上达领先性能，推理延迟低且内存占用可控。
- 适合需要实时响应的真实场景导航系统开发。
真实场景下的视觉语言导航（VLN）要求智能体处理连续视觉流，并基于语言指令生成低延迟动作。尽管视频大语言模型（Video-LLMs）推动了该领域进展，但现有基于Video-LLM的方法常面临细粒度视觉理解、长时上下文建模与计算效率之间的权衡。本文提出StreamVLN，一种流式VLN框架，采用混合慢-快上下文建模策略，支持对交错的视觉、语言和动作输入进行多模态推理。快速流通过滑动窗口的多轮对话实现响应式动作生成，而慢速更新的记忆流则利用3D感知的标记剪枝策略压缩历史视觉状态。该慢-快设计通过键值缓存复用实现实时对话，支持长视频流且保持上下文大小与推理成本恒定。在VLN-CE基准上的实验表明，StreamVLN达到当前最优性能，兼具低延迟、鲁棒性与高效性，适用于真实部署。项目主页：https://streamvln.github.io/
原文摘要 · Abstract (English)
Vision-and-Language Navigation (VLN) in real-world settings requires agents to process continuous visual streams and generate actions with low latency grounded in language instructions. While Video-based Large Language Models (Video-LLMs) have driven recent progress, current VLN methods based on Video-LLM often face trade-offs among fine-grained visual understanding, long-term context modeling and computational efficiency. We introduce StreamVLN, a streaming VLN framework that employs a hybrid slow-fast context modeling strategy to support multi-modal reasoning over interleaved vision, language and action inputs. The fast-streaming dialogue context facilitates responsive action generation through a sliding-window of multi-turn dialogues, while the slow-updating memory context compresses historical visual states using a 3D-aware token pruning strategy. With this slow-fast design, StreamVLN achieves real-time dialogues through KV cache reuse, supporting long video streams with bounded context size and inference cost. Experiments on VLN-CE benchmarks show state-of-the-art performance with low latency, ensuring robustness and efficiency in real-world deployment. The project page is: https://streamvln.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。