提出新算法，让强化学习在不确定环境中更稳健地做出风险规避决策。
Efficient Algorithms for Mitigating Uncertainty and Risk in Reinforcement Learning
- 基于坐标上升动态规划，迭代优化模型权重以提升策略性能。
- 证明了指数期望风险度量下最优策略的存在性，并给出收敛算法。
- 首次实现无模型风险规避强化学习，确保算法收敛到最优解。
本论文提出三项核心贡献：首先，发现策略梯度与动态规划在马尔可夫模型不确定性问题（MMDPs）中的新关联，提出坐标上升动态规划（CADP）算法，通过迭代调整模型权重，求解平均不确定模型下的最优马尔可夫策略，保证单调策略改进至局部最优；其次，建立指数期望风险最小化（ERM）贝尔曼算子为压缩映射的充要条件，证明了ERM-TRC与EVaR-TRC存在平稳确定性最优策略，并分别提出指数值迭代、策略迭代和线性规划算法用于计算该最优策略；最后，针对无模型场景，提出基于Q-learning的风险规避算法，解决传统方法中贝尔曼算子非压缩的问题，利用其单调性严格证明了算法收敛至最优风险规避值函数，从而获得最优平稳策略。
原文摘要 · Abstract (English)
This dissertation makes three main contributions. First, We identify a new connection between policy gradient and dynamic programming in MMDPs and propose the Coordinate Ascent Dynamic Programming (CADP) algorithm to compute a Markov policy that maximizes the discounted return averaged over the uncertain models. CADP adjusts model weights iteratively to guarantee monotone policy improvements to a local maximum. Second, We establish sufficient and necessary conditions for the exponential ERM Bellman operator to be a contraction and prove the existence of stationary deterministic optimal policies for ERM-TRC and EVaR-TRC. We also propose exponential value iteration, policy iteration, and linear programming algorithms for computing optimal stationary policies for ERM-TRC and EVaR-TRC. Third, We propose model-free Q-learning algorithms for computing policies with risk-averse objectives: ERM-TRC and EVaR-TRC. The challenge is that Q-learning ERM Bellman may not be a contraction. Instead, we use the monotonicity of Q-learning ERM Bellman operators to derive a rigorous proof that the ERM-TRC and the EVaR-TRC Q-learning algorithms converge to the optimal risk-averse value functions. The proposed Q-learning algorithms compute the optimal stationary policy for ERM-TRC and EVaR-TRC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。