让行为基础模型在动态变化时仍能零样本适配,提升机器人等场景的实用性。
Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamics
- 用Transformer构建信念估计器,增强对动态变化的感知能力。
- 在动态变化环境下,零样本回报最高提升2倍,覆盖离散与连续任务。
- 通过动态特异性聚类,实现对未知动态的高效泛化,适合真实世界部署。
行为基础模型(BFMs)在零样本条件下可生成任意任务策略,无需测试时训练或任务微调。其中基于无监督学习任务无关离线数据的后继度量估计方法尤为有前景。然而,这些方法无法应对动态变化,在部分可观测或转移函数改变时表现不佳,限制了其在机器人等真实场景的应用。本文指出,前向-后向(FB)表示法无法区分不同动态,导致参数化不同策略的潜在方向产生干扰。为此,提出基于Transformer的信念估计器改进的FB模型,显著提升零样本适应能力;同时,将策略编码空间按动态特性聚类,与上下文嵌入方向对齐,进一步提升性能。实验表明,在动态变化设置下,该方法在离散与连续任务中均实现比基线高出最多2倍的零样本回报。
原文摘要 · Abstract (English)
Behavioral Foundation Models (BFMs) proved successful in producing policies for arbitrary tasks in a zero-shot manner, requiring no test-time training or task-specific fine-tuning. Among the most promising BFMs are the ones that estimate the successor measure learned in an unsupervised way from task-agnostic offline data. However, these methods fail to react to changes in the dynamics, making them inefficient under partial observability or when the transition function changes. This hinders the applicability of BFMs in a real-world setting, e.g., in robotics, where the dynamics can unexpectedly change at test time. In this work, we demonstrate that Forward-Backward (FB) representation, one of the methods from the BFM family, cannot distinguish between distinct dynamics, leading to an interference among the latent directions, which parametrize different policies. To address this, we propose a FB model with a transformer-based belief estimator, which greatly facilitates zero-shot adaptation. We also show that partitioning the policy encoding space into dynamics-specific clusters, aligned with the context-embedding directions, yields additional gain in performance. These traits allow our method to respond to the dynamics observed during training and to generalize to unseen ones. Empirically, in the changing dynamics setting, our approach achieves up to a 2x higher zero-shot returns compared to the baselines for both discrete and continuous tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。