用图像扩散模型生成视觉路径,让机器人按自然语言指令自主导航。
VENTURA: Adapting Image Diffusion Models for Unified Task Conditioned Navigation
- 用扩散模型生成图像空间的路径掩码,实现细粒度语义导航。
- 真实场景下任务成功率提升33%,碰撞率降低54%。
- 支持未见任务组合,具备组合式推理能力,适合开放环境机器人导航。
机器人需适应多样人类指令,并在非结构化开放环境中安全运行。当前视觉语言模型虽具强语义感知先验,但因动作空间与预训练目标差异,难以直接用于导航。为此,我们提出VENTURA,一个通过微调互联网预训练图像扩散模型实现路径规划的视觉语言导航系统。不同于直接预测低级动作,VENTURA在图像空间生成路径掩码(即视觉计划),捕捉细粒度、上下文感知的导航行为。轻量级行为克隆策略将这些视觉计划转化为可执行轨迹,实现根据自然语言指令生成多样化机器人行为。为规模化训练,我们采用自监督追踪模型生成的路径掩码与VLM增强的描述进行监督,避免人工像素级标注或复杂数据采集。在大量真实世界评估中,VENTURA在物体抓取、障碍物避让和地形偏好任务上优于现有基线模型,成功率达33%提升,碰撞率下降54%,且在已见与未见场景中均表现优异。尤其发现其能泛化至未见的任务组合,展现涌现的组合能力。
原文摘要 · Abstract (English)
Robots must adapt to diverse human instructions and operate safely in unstructured, open-world environments. Recent Vision-Language models (VLMs) offer strong priors for grounding language and perception, but remain difficult to steer for navigation due to differences in action spaces and pretraining objectives that hamper transferability to robotics tasks. Towards addressing this, we introduce VENTURA, a vision-language navigation system that finetunes internet-pretrained image diffusion models for path planning. Instead of directly predicting low-level actions, VENTURA generates a path mask (i.e. a visual plan) in image space that captures fine-grained, context-aware navigation behaviors. A lightweight behavior-cloning policy grounds these visual plans into executable trajectories, yielding an interface that follows natural language instructions to generate diverse robot behaviors. To scale training, we supervise on path masks derived from self-supervised tracking models paired with VLM-augmented captions, avoiding manual pixel-level annotation or highly engineered data collection setups. In extensive real-world evaluations, VENTURA outperforms state-of-the-art foundation model baselines on object reaching, obstacle avoidance, and terrain preference tasks, improving success rates by 33% and reducing collisions by 54% across both seen and unseen scenarios. Notably, we find that VENTURA generalizes to unseen combinations of distinct tasks, revealing emergent compositional capabilities. Videos, code, and additional materials: https://venturapath.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。