通过数据集提取任务向量，提升零样本离线强化学习性能
Improving Zero-Shot Offline RL via Behavioral Task Sampling
- 从离线数据中提取任务向量，替代随机采样
- 在多个基准环境上平均提升零样本性能20%
- 适用于需要快速适配新奖励函数的离线强化学习场景
离线零样本强化学习旨在不进行额外环境交互的情况下，训练出能优化未知奖励函数的智能体。现有方法通过采样定义线性奖励函数的任务向量来训练任务条件策略。大多数算法随机采样任务向量，隐含假设其可充分覆盖任务空间。我们指出这会导致零样本泛化性能下降。为此，我们提出直接从离线数据集中提取任务向量，并用于定义策略训练的任务分布。设计了一种简单通用的奖励函数提取方法，可集成至现有离线零样本RL算法。在多个基准环境与基线上的实验表明，该方法平均提升零样本性能20%，凸显了合理任务采样在离线零样本强化学习中的重要性。
原文摘要 · Abstract (English)
Offline zero-shot reinforcement learning (RL) aims to learn agents that optimize unseen reward functions without additional environment interaction. The standard approach to this problem trains task-conditioned policies by sampling task vectors that define linear reward functions over learned state representations. In most existing algorithms, these task vectors are randomly sampled, implicitly assuming this adequately captures the structure of the task space. We argue that doing so leads to suboptimal zero-shot generalization. To address this limitation, we propose extracting task vectors directly from the offline dataset and using them to define the task distribution used for policy training. We introduce a simple and general reward function extraction procedure that integrates into existing offline zero-shot RL algorithms. Across multiple benchmark environments and baselines, our approach improves zero-shot performance by an average of 20%, highlighting the importance of principled task sampling in offline zero-shot RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。