通过测试时微调让视觉语言模型实现零样本价值函数,提升泛化与时间推理能力。
VITA: Zero-Shot Value Functions via Test-Time Adaptation of Vision-Language Models
- 测试时通过轻量模块梯度更新,动态优化价值估计。
- 在真实机器人任务中实现单一环境训练跨分布泛化,性能超越现有最优方法。
- 适用于零样本奖励塑造,可生成多任务强化学习策略。
视觉语言模型(VLM)作为零样本目标条件价值函数展现出潜力,但其冻结的预训练表征限制了泛化能力和时间推理。我们提出VITA,一种通过测试时适应增强这两方面能力的零样本价值函数学习方法。推理时,一个轻量级适应模块通过元学习的自监督损失进行梯度更新,每次测试时更新均改进价值估计。通过沿轨迹顺序更新,VITA将历史信息编码至参数中,解决时间推理局限。为缓解捷径学习,我们提出基于差异性的采样策略,在训练中选择语义多样化的轨迹片段。在真实世界机器人操作任务中,VITA可从单一训练环境泛化到多种分布外的任务、环境和执行体,性能优于使用自回归VLM的现有最先进零样本方法。此外,我们证明VITA的零样本价值估计可用于离线强化学习中的奖励塑形,在Meta-World基准上生成的多任务策略,表现超过使用仿真中模糊逻辑密集奖励训练的策略。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) show promise as zero-shot goal-conditioned value functions, but their frozen pre-trained representations limit generalization and temporal reasoning. We introduce VITA, a zero-shot value function learning method that enhances both capabilities via test-time adaptation. At inference, a lightweight adaptation module is updated via a gradient step on a meta-learned self-supervised loss, such that each test-time update improves value estimation. By updating sequentially over a trajectory, VITA encodes history into its parameters, addressing the temporal reasoning limitations. To mitigate shortcut learning, we propose a dissimilarity-based sampling strategy that selects semantically diverse segments of the trajectory during training. In real-world robotic manipulation tasks, VITA generalizes from a single training environment to diverse out-of-distribution tasks, environments, and embodiments, outperforming the state-of-the-art zero-shot method using autoregressive VLMs. Furthermore, we demonstrate that VITA's zero-shot value estimates can be utilized for reward shaping in offline reinforcement learning, resulting in multi-task policies on the Meta-World benchmark that exceed the performance of those trained with the simulation's fuzzy-logic dense rewards. Project website: https://chziakas.github.io/vita/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。