提出新型量化Q-learning算法，实现风险敏感强化学习的收敛与高效优化。
Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis
- 基于新动态规划分解框架，无需已知转移概率即可实现量化MDP求解。
- 在表格环境中验证算法收敛性，性能优于已有方法。
- 适合关注风险控制的强化学习研究者与实际应用开发者。
在马尔可夫决策过程（MDPs）中，分位数风险度量（如风险价值）是建模强化学习智能体对特定结果偏好性的标准指标。本文提出一种新的针对量化优化的Q-learning算法，具备强收敛性与性能保证。该算法利用一种新颖且简单的动态规划（DP）分解方法来处理量化MDPs。相较于先前工作，该分解方法无需已知转移概率，也无需求解复杂的鞍点方程，为其他无模型强化学习算法提供了良好基础。在表格化环境中的数值实验表明，所提Q-learning算法能收敛至其对应的DP变体，并在性能上超越早期算法。
原文摘要 · Abstract (English)
In Markov decision processes (MDPs), quantile risk measures such as Value-at-Risk are a standard metric for modeling RL agents' preferences for certain outcomes. This paper proposes a new Q-learning algorithm for quantile optimization in MDPs with strong convergence and performance guarantees. The algorithm leverages a new, simple dynamic program (DP) decomposition for quantile MDPs. Compared with prior work, our DP decomposition requires neither known transition probabilities nor solving complex saddle point equations and serves as a suitable foundation for other model-free RL algorithms. Our numerical results in tabular domains show that our Q-learning algorithm converges to its DP variant and outperforms earlier algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。