用检索代替微调,让视觉语言动作模型在测试时低成本扩展新任务
Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time

- 通过检索预存的示范数据,无需微调即可适应新任务
- 在PushT和RoboTwin 2.0上实现跨形态泛化,真实机器人验证成功
- 适合需要快速部署新任务的机器人系统,尤其适配视频生成型世界动作模型
将视觉语言动作(VLA)策略扩展至新任务通常需特定任务的遥操作示范和逐任务微调,导致数据收集与计算成本高昂。本文表明,这种目标端的逐任务适应成本可被检索替代。所提检索增强策略仅需一次训练:在目标体感(查询)与更低成本体感(如人手视频,作为池)的配对示范上训练,之后冻结参数。新任务在部署时通过向检索池添加池侧示范实现。冻结策略在每个控制步骤依赖检索到的轨迹,使新任务通过数据索引而非参数更新吸收。仅当引入全新未见体感时才需微调。实验显示,该方法在多种骨干网络上均有效,尤其在基于视频生成的世界动作模型(Cosmos Policy)中表现突出。此时,检索提供粗粒度任务进展,而模型的未来图像目标则提供视觉一致性信号,强化检索条件下的动作。在PushT上,检索为跨体感泛化至未见目标角度提供了可复用的高层运动先验;在RoboTwin 2.0上,方法优于跨体感基线,在真实机器人上也成功演示。
原文摘要 · Abstract (English)
Extending a vision-language-action (VLA) policy to a new task typically requires task-specific teleoperated demonstrations and per-task fine-tuning, making adaptation costly in both data collection and compute. In this paper, we show that this target-side per-task adaptation cost can be replaced by retrieval. Our retrieval-augmented policy is trained once on paired demonstrations from the target embodiment (query) and a cheaper embodiment (pool, e.g., human-hand video), then frozen. New tasks are added at deployment by appending pool-side demonstrations to a retrieval pool. The frozen policy conditions on retrieved trajectories at every control step, so new tasks are absorbed by indexing data rather than updating parameters. Fine-tuning is needed only to take on a new, unseen embodiment, not for each new task. We show that retrieval improves policies beyond a specific backbone, including standard VLA policies, but its effect is especially pronounced in Cosmos Policy, a video-generation-based world-action model (WAM). In this setting, retrieval supplies coarse task progression, while the WAM's future-image objective provides an additional visual consistency signal that strengthens the retrieval-conditioned actions. On PushT, we study how retrieval provides a reusable high-level motion prior for cross-embodiment generalization to unseen goal angles, while on RoboTwin 2.0 our method outperforms cross-embodiment baselines on unseen tasks, and we additionally demonstrate the method on a real robot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。