用特征向量方法改进平均奖励强化学习，提升稳定性和收敛速度。
EVAL: EigenVector-based Average-reward Learning
- 基于特征向量构建神经网络近似框架，线性求解最优策略与平均奖励。
- 在经典控制任务中，收敛速度和稳定性优于现有算法。
- 适用于无熵正则化场景，理论联系更紧密，适合研究者参考。
在强化学习中，折扣奖励和平均奖励是两类广泛研究的目标函数。将熵正则化引入这两种目标，显著提升了鲁棒性和探索能力。近期，针对熵正则化的平均奖励问题，已有研究在表格设置下利用大偏差理论解决，该方法具有线性特性，仅通过单一矩阵即可获得最优策略与平均奖励率。本文将该框架扩展至更一般场景，提出基于神经网络函数逼近的方法。该形式揭示了不同强化学习目标间的深层理论关联。此外，我们结合后验策略迭代方案，证明该方法亦可解决无熵正则化的平均奖励问题。在经典控制基准测试中，实验表明该方法在稳定性与收敛速率方面优于其他算法。
原文摘要 · Abstract (English)
In reinforcement learning, two objective functions have been developed extensively in the literature: discounted and averaged rewards. The generalization to an entropy-regularized setting has led to improved robustness and exploration for both of these objectives. Recently, the entropy-regularized average-reward problem was addressed using tools from large deviation theory in the tabular setting. This method has the advantage of linearity, providing access to both the optimal policy and average reward-rate through properties of a single matrix. In this paper, we extend that framework to more general settings by developing approaches based on function approximation by neural networks. This formulation reveals new theoretical insights into the relationship between different objectives used in RL. Additionally, we combine our algorithm with a posterior policy iteration scheme, showing how our approach can also solve the average-reward RL problem without entropy-regularization. Using classic control benchmarks, we experimentally find that our method compares favorably with other algorithms in terms of stability and rate of convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。