通过自回归特征与优势加权,提升行为基础模型的泛化与精度。
Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weighting
- 引入自回归特征打破任务编码线性限制,支持非线性表达。
- 结合离线强化学习技术,在DMC Humanoid等数据集上实现稳定性能。
- 在复杂空间任务中表现更优,适合需要精细泛化的场景。
前向-后向表示(FB)是一种新提出的框架,用于训练行为基础模型(BFM),旨在无需为每个新任务重新训练,即可在给定强化学习环境中提供零样本高效策略。本文针对FB训练的两个核心局限进行改进。首先,如所有基于后续特征的方法一样,FB依赖于任务的线性编码:测试时,每个新奖励函数被线性投影到一组预训练特征上,这限制了任务表征的表达能力与精度。为此,本文引入自回归特征,使细粒度任务特征依赖于粗粒度任务信息,从而支持任意非线性任务编码,显著提升FB框架的表达能力。其次,离线数据集训练强化学习代理通常需特定技术。本文证明FB可与这类离线强化学习技术兼容,通过适配Nair et al. (2020b) 和 Cetin et al. (2024) 的方法,解决部分数据集(如DMC Humanoid)中的性能退化问题。结果表明,所提出的离线FB BFM在多个新环境中表现优异;在D4RL运动基准测试中,通用FB代理性能达到标准单任务离线代理(IQL、XQL)水平。在多数设置中,离线技术是获得良好性能的必要条件。自回归特征对性能有正向影响,但幅度适中,主要体现在需要空间精度和超出训练行为集泛化的任务中。
原文摘要 · Abstract (English)
The forward-backward representation (FB) is a recently proposed framework (Touati et al., 2023; Touati & Ollivier, 2021) to train behavior foundation models (BFMs) that aim at providing zero-shot efficient policies for any new task specified in a given reinforcement learning (RL) environment, without training for each new task. Here we address two core limitations of FB model training. First, FB, like all successor-feature-based methods, relies on a linear encoding of tasks: at test time, each new reward function is linearly projected onto a fixed set of pre-trained features. This limits expressivity as well as precision of the task representation. We break the linearity limitation by introducing auto-regressive features for FB, which let finegrained task features depend on coarser-grained task information. This can represent arbitrary nonlinear task encodings, thus significantly increasing expressivity of the FB framework. Second, it is well-known that training RL agents from offline datasets often requires specific techniques.We show that FB works well together with such offline RL techniques, by adapting techniques from (Nair et al.,2020b; Cetin et al., 2024) for FB. This is necessary to get non-flatlining performance in some datasets, such as DMC Humanoid. As a result, we produce efficient FB BFMs for a number of new environments. Notably, in the D4RL locomotion benchmark, the generic FB agent matches the performance of standard single-task offline agents (IQL, XQL). In many setups, the offline techniques are needed to get any decent performance at all. The auto-regressive features have a positive but moderate impact, concentrated on tasks requiring spatial precision and task generalization beyond the behaviors represented in the trainset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。