用文本生成图像增强视觉导航，让智能体更准找到目标
Do Visual Imaginations Improve Vision-and-Language Navigation Agents?
- 用扩散模型根据指令生成关键地标图像作为导航辅助
- 平均提升1%成功率，路径效率指标最高提升0.5%
- 适合想提升视觉语言导航性能的研究者和开发者
视觉-语言导航（VLN）智能体需根据自然语言指令在未见过的环境中导航。本文研究指令中隐含的子目标视觉表征能否作为导航线索并提升性能。我们利用文本到图像扩散模型，基于分割后指令中的地标参考信息生成视觉想象，并将其作为额外模态提供给VLN智能体，同时添加辅助损失以显式关联这些图像与对应指代表达。实验结果显示，该方法在各类智能体上使成功率（SR）平均提升约1个百分点，成功度量（SPL）最高提升0.5个百分点。结果表明，相比仅依赖语言指令，该方法增强了视觉理解能力。代码与数据详见 https://www.akhilperincherry.com/VLN-Imagine-website/。
原文摘要 · Abstract (English)
Vision-and-Language Navigation (VLN) agents are tasked with navigating an unseen environment using natural language instructions. In this work, we study if visual representations of sub-goals implied by the instructions can serve as navigational cues and lead to increased navigation performance. To synthesize these visual representations or imaginations, we leverage a text-to-image diffusion model on landmark references contained in segmented instructions. These imaginations are provided to VLN agents as an added modality to act as landmark cues and an auxiliary loss is added to explicitly encourage relating these with their corresponding referring expressions. Our findings reveal an increase in success rate (SR) of around 1 point and up to 0.5 points in success scaled by inverse path length (SPL) across agents. These results suggest that the proposed approach reinforces visual understanding compared to relying on language instructions alone. Code and data for our work can be found at https://www.akhilperincherry.com/VLN-Imagine-website/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。