用大模型重写训练数据，让导航模型在没见过的环境里也能走得好。
Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation
- 用视觉语言模型和大语言模型重写观察描述，生成新场景。
- 在多个数据集上提升导航性能，尤其在未见环境中效果显著。
- 无需模拟器或人工标注，适合想提升模型泛化能力的研究者。
视觉-语言导航（VLN）领域长期受数据稀缺制约，严重限制智能体在未见环境中的泛化能力。以往方法主要依赖额外模拟器数据或网络收集的图像/视频来提升泛化性，但模拟环境多样性有限，网络数据又需大量人力清洗噪声。本文提出一种重写驱动的数据增强范式（RAM），直接通过重写人类标注的训练数据生成未见的观察-指令对。得益于重写机制，新数据可在无需模拟器且省去人工劳动的前提下获得，有效促进泛化。具体而言，我们引入物体增强型观察重写，结合视觉语言模型（VLM）与大语言模型（LLM），生成富含物体和空间布局信息的重写场景描述，并通过文本到图像生成模型（T2IMs）合成观察；随后提出观察对比式指令重写，利用LLM推理原观察与新观察的差异，生成对齐的新指令。我们进一步设计了先混合后聚焦的训练策略与随机裁剪方案，有效提升数据分布多样性并抑制增强数据噪声。在离散环境（R2R、REVERIE、R4R）和连续环境（R2R-CE）上的实验表明，该方法显著优于现有方法，具备出色的泛化能力。代码已开源：https://github.com/SaDil13/VLN-RAM。
原文摘要 · Abstract (English)
Data scarcity is a long-standing challenge in the Vision-Language Navigation (VLN) field, which extremely hinders the generalization of agents to unseen environments. Previous works primarily rely on additional simulator data or web-collected images/videos to improve the generalization. However, the simulator environments still face limited diversity, and the web-collected data often requires extensive labor to remove the noise. In this paper, we propose a Rewriting-driven AugMentation (RAM) paradigm for VLN, which directly creates the unseen observation-instruction pairs via rewriting human-annotated training data. Benefiting from our rewriting mechanism, new observation-instruction pairs can be obtained in both simulator-free and labor-saving manners to promote generalization. Specifically, we first introduce Object-Enriched Observation Rewriting, where we combine Vision-Language Models (VLMs) and Large Language Models (LLMs) to derive rewritten object-enriched scene descriptions, enabling observation synthesis with diverse objects and spatial layouts via Text-to-Image Generation Models (T2IMs). Then, we propose Observation-Contrast Instruction Rewriting, which generates observation-aligned rewritten instructions by requiring LLMs to reason the difference between original and new observations. We further develop a mixing-then-focusing training strategy with a random observation cropping scheme, effectively enhancing data distribution diversity while suppressing augmentation data noise during training. Experiments on both the discrete environments (R2R, REVERIE, and R4R datasets) and continuous environments (R2R-CE dataset) show the superior performance and impressive generalization ability of our method. Code is available at https://github.com/SaDil13/VLN-RAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。