让机器人模型学会处理没见过的物体,靠外部知识增强决策能力
VLA^2: Empowering Vision-Language-Action Models with an Agentic Framework for Unseen Concept Manipulation
- 用外部检索和检测模块为视觉语言动作模型补充未知物体信息
- 在难任务上成功率达78.3%,比基线提升44.2%
- 适合需要灵活应对新物体的机器人应用开发者
当前视觉-语言-动作(VLA)模型虽在大规模机器人数据上预训练,具备强多任务能力和良好泛化性,但在面对训练数据外的物体概念时成功率显著下降,如未见的物体描述与纹理。为此,我们提出新型代理框架VLA^2,以OpenVLA为执行核心,有效融合外部模块如网络检索与物体检测,为VLA提供目标物体的视觉与文本知识,缓解分布外物体处理中的泛化失败问题。基于LIBERO仿真环境,我们引入新物体与描述,构建包含三个难度层级的新评估基准,用于测试方法有效性。实验表明,该框架在自定义的高难度泛化基准上超越现有最先进模型;相较于独立的OpenVLA基线,在硬级基准上成功率达78.3%,提升44.2%,所有定制环境中平均提升20.2%,且不损害域内任务表现。
原文摘要 · Abstract (English)
Current vision-language-action (VLA) models, pre-trained on large-scale robotic data, exhibit strong multi-task capabilities and generalize well to variations in visual and language instructions for manipulation. However, their success rate drops significantly when faced with object concepts outside the training data, such as unseen object descriptions and textures in the dataset. To address this, we propose a novel agentic framework, VLA^2, which leverages OpenVLA as the execution backbone and effectively leverages external modules such as web retrieval and object detection to provide visual and textual knowledge about target objects to the VLA. This approach mitigates generalization failure when handling out-of-distribution objects. Based on the LIBERO simulation environment, we introduced novel objects and object descriptions to construct a new evaluation benchmark with three difficulty levels to test the effectiveness of our method. Our framework successfully outperformed the current state-of-the-art models on our designed hard-level generalization benchmark. Compared to the standalone OpenVLA baseline, VLA^2 achieves a 44.2% improvement in the success rate in the hard-level benchmark and an average improvement of 20.2% in all customized environments without any performance degradation on in-domain tasks. Project website: https://vla-2.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。