用递归奖励聚合让智能体自动对齐复杂目标，无需改奖励函数。
Recursive Reward Aggregation
- 通过代数视角重构马尔可夫决策过程，让贝尔曼方程自然衍生出递归奖励聚合。
- 支持折扣最大值、夏普比率等新型聚合方式，在多种任务上优化效果优于传统方法。
- 适用于确定性和随机环境，适合想灵活调整智能体目标的研究者与开发者。
在强化学习中，使智能体行为与特定目标对齐通常需要精心设计奖励函数，但当目标复杂时这很困难。本文提出一种无需修改奖励函数的灵活行为对齐方法，通过选择合适的奖励聚合函数实现。从马尔可夫决策过程的代数视角出发，我们证明贝尔曼方程可自然由奖励的递归生成与聚合导出，从而将标准的折扣求和推广至其他递归聚合形式，如折扣最大值和夏普比率。该方法适用于确定性与随机场景，并能无缝集成到基于价值和演员-评论家的算法中。实验表明，该方法能有效优化多样化的目标，凸显其通用性与实际应用潜力。
原文摘要 · Abstract (English)
In reinforcement learning (RL), aligning agent behavior with specific objectives typically requires careful design of the reward function, which can be challenging when the desired objectives are complex. In this work, we propose an alternative approach for flexible behavior alignment that eliminates the need to modify the reward function by selecting appropriate reward aggregation functions. By introducing an algebraic perspective on Markov decision processes (MDPs), we show that the Bellman equations naturally emerge from the recursive generation and aggregation of rewards, allowing for the generalization of the standard discounted sum to other recursive aggregations, such as discounted max and Sharpe ratio. Our approach applies to both deterministic and stochastic settings and integrates seamlessly with value-based and actor-critic algorithms. Experimental results demonstrate that our approach effectively optimizes diverse objectives, highlighting its versatility and potential for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。