通过操控Q函数实现高效奖励迁移，无需重新学习即可快速适应新目标。
Provably Efficient Reward Transfer in Reinforcement Learning with Discrete Markov Decision Processes
- 基于源奖励函数的已知组合关系，直接操作Q函数边界
- 在学习前就通过边界剪枝减少搜索空间，提升效率
- 适用于有轻量辅助模型的离散马尔可夫决策问题
本文提出一种新的强化学习中奖励适应（RA）方法。当智能体需在相同环境动态下适应新目标奖励函数时，若已有多个不同奖励下预先学习的源行为，可利用这些信息避免从头学习。我们通过操纵Q函数，假设目标奖励是源奖励的已知函数，推导出Q函数的上下界，并设计一种类似值迭代的迭代过程来不断收紧这些边界。这些边界可在学习开始前用于动作剪枝，从而显著降低探索开销。该方法称为“Q-M”（Q-Manipulation）。算法依赖一个轻量级辅助模型，易于获取或训练。我们证明，在离散域下，该方法不破坏最优策略，且在概率意义上具有可证明的样本效率。在多种合成与仿真环境中验证了其有效性、泛化性与实用性。
原文摘要 · Abstract (English)
In this paper, we propose a new solution to reward adaptation (RA) in reinforcement learning, where the agent adapts to a target reward function based on one or more existing source behaviors learned a priori under the same domain dynamics but different reward functions. While learning the target behavior from scratch is possible, it is often inefficient given the available source behaviors. Our work introduces a new approach to RA through the manipulation of Q-functions. Assuming the target reward function is a known function of the source reward functions, we compute bounds on the Q-function and present an iterative process (akin to value iteration) to tighten these bounds. Such bounds enable action pruning in the target domain before learning even starts. We refer to this method as "Q-Manipulation" (Q-M). The iteration process assumes access to a lite-model, which is easy to provide or learn. We formally prove that Q-M, under discrete domains, does not affect the optimality of the returned policy and show that it is provably efficient in terms of sample complexity in a probabilistic sense. Q-M is evaluated in a variety of synthetic and simulation domains to demonstrate its effectiveness, generalizability, and practicality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。