提出可重构物理测试平台BusyBox,评估视觉语言动作模型对新物体的物理操作泛化能力。
Benchmarking Affordance Generalization with BusyBox
- 设计可更换旋转模块的物理装置,保持功能一致但外观多变
- 强模型在变体任务上仍表现不佳,证明泛化难度大
- 开源全套3D打印文件和数据集,方便实验室复现
视觉-语言-动作(VLA)模型因其泛化潜力受到广泛关注。尽管单任务策略仍具竞争力,但VLA正日益能处理训练中未见的任务与环境。在视觉和语言空间的泛化固然重要,但更关键的是物理操作泛化——即对具有熟悉物理特征的新物体进行操作的能力。本文提出BusyBox,一个用于系统性半自动评估VLA物理操作泛化能力的物理基准。该基准包含6个模块:开关、滑块、电线、按钮、显示屏和旋钮,可通过互换与旋转生成多种外观不同但功能相同的变体。我们实证表明,即使是强大开放权重的VLA如$π_{0.5}$和GR00T-N1.6,在跨变体任务上仍面临巨大挑战。为促进社区评估与实验,我们设计了可在多数机器人实验室搭建的系统,并发布全部3D打印零件的CAD文件及电子元件清单。同时公开使用双臂移动式Mobile Aloha机器人在标准配置下收集的语言标注示范数据集。所有材料均在https://microsoft.github.io/BusyBox发布。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have been attracting the attention of researchers and practitioners thanks to their promise of generalization. Although single-task policies still offer competitive performance, VLAs are increasingly able to handle commands and environments unseen in their training set. While generalization in vision and language space is undoubtedly important for robust versatile behaviors, a key meta-skill VLAs need to possess is affordance generalization -- the ability to manipulate new objects with familiar physical features. In this work, we present BusyBox, a physical benchmark for systematic semi-automatic evaluation of VLAs' affordance generalization. BusyBox consists of 6 modules with switches, sliders, wires, buttons, a display, and a dial. The modules can be swapped and rotated to create a multitude of BusyBox variations with different visual appearances but the same set of affordances. We empirically demonstrate that generalization across BusyBox variants is highly challenging even for strong open-weights VLAs such as $π_{0.5}$ and GR00T-N1.6. To encourage the research community to evaluate their own VLAs on BusyBox and to propose new affordance generalization experiments, we have designed BusyBox to be easy to build in most robotics labs. We release the full set of CAD files for 3D-printing its parts as well as a bill of materials for (optionally) assembling its electronics. We also publish a dataset of language-annotated demonstrations that we collected using the common bimanual Mobile Aloha robot on the canonical BusyBox configuration. All of the released materials are available at https://microsoft.github.io/BusyBox.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。