提出一种改进的连续动作强化学习算法，通过限制优势函数提升性能。
Mirror Descent Actor Critic via Bounded Advantage Learning
- 用有界对数密度项改进价值函数损失，增强学习稳定性
- 在多个环境上优于无正则化和仅熵正则化的方法
- 理论分析支持有界优势项的有效性，适合连续控制任务
正则化是近期强化学习算法的核心组件。镜面下降值迭代（MDVI）在值函数和策略更新中同时使用相对熵与熵作为正则化项。尽管其在离散动作域表现出色且具备强理论保证，但在连续动作域中，基于KL-熵正则化的性能仍不及仅使用熵正则化的强基线方法。本文提出镜面下降演员-评论家（MDAC），作为适用于连续动作域的MDVI的演员-评论家形式实例，并证明在评论家损失函数中对演员的对数密度项施加边界约束后，其性能显著优于未加约束的朴素实现。进一步地，我们通过回顾在表格情形下演员对数概率等价于正则化优势函数，理论上探讨了何时及为何对优势项进行约束是合理且有益的。我们还实证探索了有效的边界函数选择，结果表明，在适当选择边界函数时，MDAC性能优于强非正则化及仅熵正则化方法。
原文摘要 · Abstract (English)
Regularization is a core component of recent Reinforcement Learning (RL) algorithms. Mirror Descent Value Iteration (MDVI) uses both Kullback-Leibler divergence and entropy as regularizers in its value and policy updates. Despite its empirical success in discrete action domains and strong theoretical guarantees, the performance of KL-entropy-regularized methods does not surpass that of a strong entropy-only-regularized method in continuous action domains. In this study, we propose Mirror Descent Actor Critic (MDAC) as an actor-critic style instantiation of MDVI for continuous action domains, and show that its empirical performance is significantly boosted by bounding the actor's log-density terms in the critic's loss function, compared to a non-bounded naive instantiation. Further, we relate MDAC to Advantage Learning by recalling that the actor's log-probability is equal to the regularized advantage function in tabular cases, and theoretically discuss when and why bounding the advantage terms is validated and beneficial. We also empirically explore effective choices for the bounding functions, and show that MDAC performs better than strong non-regularized and entropy-only-regularized methods with an appropriate choice of the bounding functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。