用视觉语言模型提前分析环境,自动调整物体位置避免机器人执行失败。
StageCraft: Execution Aware Mitigation of Distractor and Obstruction Failures in VLA Models
- 通过视觉语言模型推理初始环境,主动调整物体位置
- 在真实场景中使VLA模型性能提升40%以上
- 无需修改模型,仅需少量演示视频即可使用
大规模文本与图像预训练结合多样机器人示范,使视觉语言动作模型(VLAs)能泛化至新任务、新物体和新场景。然而,这些模型在执行时仍易受干扰物和物理遮挡影响而失败。现有策略通过微调提升泛化能力,但在未见干扰场景下仍表现不佳。本文探究是否可利用互联网规模的视觉语言模型(VLM)预训练能力,来推理并缓解此类执行失败。为此,提出StageCraft——一种无需训练的方法:基于政策回放视频与成功标签,利用VLM的上下文推理能力,判断初始状态中哪些物体需调整以避免预期失败。该方法为可扩展的即插即用模块,不增加底层策略约束,仅需少量策略回放即可运行。在三个包含多样化干扰物与遮挡的真实任务领域中评估,StageCraft使主流VLA模型性能绝对提升40%。仿真实验在RLBench中显示,其干预强度随基础策略能力自适应调节,并随着更多上下文样本提升效果。相关视频见https://stagecraft-decorator.github.io/stagecraft/。
原文摘要 · Abstract (English)
Large scale pre-training on text and image data along with diverse robot demonstrations has helped Vision Language Action models (VLAs) to generalize to novel tasks, objects and scenes. However, these models are still susceptible to failure in the presence of execution-time impediments such as distractors and physical obstructions in the robot's workspace. Existing policy improvement methods finetune base VLAs to improve generalization, yet they still struggle in unseen distractor settings. To address this problem, we investigate whether internet-scale pretraining of large vision-language models (VLMs) can be leveraged to reason about these impediments and mitigate policy failures. To this end, we propose StageCraft, a training-free approach to improve pretrained VLA policy performance by manipulating the environment's initial state using VLM-based in-context reasoning. StageCraft takes policy rollout videos and success labels as input and leverages VLM's reasoning ability to infer which objects in the initial state need to be manipulated to avoid anticipated execution failures. StageCraft is an extensible plug-and-play module that does not introduce additional constraints on the underlying policy, and only requires a few policy rollouts to work. We evaluate performance of state-of-the-art VLA models with StageCraft and show an absolute 40% performance improvement across three real world task domains involving diverse distractors and obstructions. Our simulation experiments in RLBench empirically show that StageCraft tailors its extent of intervention based on the strength of the underlying policy and improves its performance with more in-context samples. Videos of StageCraft in effect can be found at https://stagecraft-decorator.github.io/stagecraft/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。