从最优策略中高效恢复随时间变化的奖励函数，利用稀疏性和低秩先验。
Efficient Reward Identification In Max Entropy Reinforcement Learning with Sparsity and Rank Priors
- 用线性约束下的稀疏化建模奖励常数且变化少的特性
- 提出多项式时间算法，精确求解稀疏奖励恢复问题
- 适用于需要简化奖励结构的强化学习应用
本文研究从最大熵强化学习中的最优策略或示范数据中恢复随时间变化的奖励函数问题。该问题在缺乏额外假设时高度不适定。但在许多实际场景中，奖励本身具有稀疏性，且存在先验信息。本文考虑两种先验：1）奖励多数时间恒定且变化不频繁；2）奖励可由少量特征函数的线性组合表示。我们证明第一种先验可转化为带线性约束的稀疏化问题，并给出多项式时间精确求解算法。第二种先验可转化为带线性约束的低秩最小化问题，可通过核范数等凸松弛方法处理。上述发现均导出高效的基于优化的奖励识别算法。多个实验验证了恢复奖励的准确性及其泛化能力。
原文摘要 · Abstract (English)
In this paper, we consider the problem of recovering time-varying reward functions from either optimal policies or demonstrations coming from a max entropy reinforcement learning problem. This problem is highly ill-posed without additional assumptions on the underlying rewards. However, in many applications, the rewards are indeed parsimonious, and some prior information is available. We consider two such priors on the rewards: 1) rewards are mostly constant and they change infrequently, 2) rewards can be represented by a linear combination of a small number of feature functions. We first show that the reward identification problem with the former prior can be recast as a sparsification problem subject to linear constraints. Moreover, we give a polynomial-time algorithm that solves this sparsification problem exactly. Then, we show that identifying rewards representable with the minimum number of features can be recast as a rank minimization problem subject to linear constraints, for which convex relaxations of rank can be invoked. In both cases, these observations lead to efficient optimization-based reward identification algorithms. Several examples are given to demonstrate the accuracy of the recovered rewards as well as their generalizability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。