用语言生成想象替代视觉生成,让导航模型更高效准确。
Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation
- 用大模型双分支架构,分别负责逻辑推理和场景想象
- 在R2R数据集上达到顶尖性能,参数量更少
- 适合需要轻量化高效导航的智能机器人应用
视觉-语言导航(VLN)要求智能体在部分可观测环境下根据自然语言指令进行导航,难以对齐感知与语言。现有方法通过生成未来场景缓解此问题,但依赖视觉合成,计算成本高且冗余信息多。为此,我们提出自适应文本幻觉器(ATD),通过语言形式动态生成关键环境语义,实现更可靠高效的导航策略。ATD采用类人左右脑架构:左脑负责逻辑整合,右脑负责未来场景想象。通过仅微调双分支中的Q-former,高效激活大模型领域知识,实现导航过程中的动态推理与想象更新。此外,引入跨交互机制对齐生成结果,并注入导航专家模块,使模型联合利用大模型推理能力与导航模型专长。在R2R基准上实验表明,ATD以更少参数取得领先性能。
原文摘要 · Abstract (English)
Vision-and-Language Navigation (VLN) requires the agent to navigate by following natural instructions under partial observability, making it difficult to align perception with language. Recent methods mitigate this by imagining future scenes, yet they rely on vision-based synthesis, leading to high computational cost and redundant details. To this end, we propose to adaptively imagine key environmental semantics via \textit{language} form, enabling a more reliable and efficient strategy. Specifically, we introduce a novel Adaptive Text Dreamer (ATD), a dual-branch self-guided imagination policy built upon a large language model (LLM). ATD is designed with a human-like left-right brain architecture, where the left brain focuses on logical integration, and the right brain is responsible for imaginative prediction of future scenes. To achieve this, we fine-tune only the Q-former within both brains to efficiently activate domain-specific knowledge in the LLM, enabling dynamic updates of logical reasoning and imagination during navigation. Furthermore, we introduce a cross-interaction mechanism to regularize the imagined outputs and inject them into a navigation expert module, allowing ATD to jointly exploit both the reasoning capacity of the LLM and the expertise of the navigation model. We conduct extensive experiments on the R2R benchmark, where ATD achieves state-of-the-art performance with fewer parameters. The code is \href{https://github.com/zhangpingrui/Adaptive-Text-Dreamer}{here}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。