用基础函数表示强化学习行为空间，实现零样本策略迁移。
Proto Successor Measure: Representing the Behavior Space of an RL Agent
- 构建与策略无关的基函数，表征智能体所有可能行为
- 测试时仅需线性组合基函数，即可生成最优策略
- 无需额外环境交互，适用于任意奖励函数
智能体在探索环境后，应能将知识迁移到下游任务中而无需额外交互，即所谓“零样本学习”。现有方法多依赖任务特性或马尔可夫决策过程（MDP）结构假设。本文提出原型后续度量（Proto Successor Measure），作为动态系统中强化学习智能体所有可能行为的基础集。证明任一行为（以访问分布表示）均可通过这些与策略无关的基函数的仿射组合表示。测试时给定奖励函数，只需求解线性权重组合对应最优策略。我们推导出一种利用环境免奖励交互数据学习基函数的实用算法，并验证该方法可在不进行额外环境交互的情况下，对任意奖励函数生成最优策略。
原文摘要 · Abstract (English)
Having explored an environment, intelligent agents should be able to transfer their knowledge to most downstream tasks within that environment without additional interactions. Referred to as "zero-shot learning", this ability remains elusive for general-purpose reinforcement learning algorithms. While recent works have attempted to produce zero-shot RL agents, they make assumptions about the nature of the tasks or the structure of the MDP. We present Proto Successor Measure: the basis set for all possible behaviors of a Reinforcement Learning Agent in a dynamical system. We prove that any possible behavior (represented using visitation distributions) can be represented using an affine combination of these policy-independent basis functions. Given a reward function at test time, we simply need to find the right set of linear weights to combine these bases corresponding to the optimal policy. We derive a practical algorithm to learn these basis functions using reward-free interaction data from the environment and show that our approach can produce the optimal policy at test time for any given reward function without additional environmental interactions. Project page: https://agarwalsiddhant10.github.io/projects/psm.html.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。