提出一种风险敏感的强化学习算法，可优化累积奖励的分位数目标。
Optimistic Reinforcement Learning with Quantile Objectives
- 基于置信区间优化分位数价值函数，实现乐观探索
- 在有限时域MDP中达到理论最优的高概率后悔上界
- 适用于医疗、金融等需考虑风险的决策场景
近年来，强化学习（RL）取得了显著进展。然而，传统RL框架未考虑目标函数的风险敏感性，这在医疗和金融等领域至关重要。一种常见方法是优化累积奖励分布的特定分位数。本文提出UCB-QRL，一种针对有限时域马尔可夫决策过程（MDP）中τ-分位数目标的乐观学习算法。UCB-QRL为迭代算法：每轮先估计转移概率，再在该估计的置信球内优化分位数价值函数。我们证明，在包含S个状态、A个动作、T个周期、H个时步的回合制设置下，UCB-QRL的高概率后悔上界为$\mathcal O\left((2/κ)^{H+1}H\sqrt{SATH\log(2SATH/δ)}\right)$。其中κ>0为依赖问题的常数，反映底层MDP分位数价值的敏感性。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has achieved tremendous success in recent years. However, the classical foundations of RL do not account for the risk sensitivity of the objective function, which is critical in various fields, including healthcare and finance. A popular approach to incorporate risk sensitivity is to optimize a specific quantile of the cumulative reward distribution. In this paper, we develop UCB-QRL, an optimistic learning algorithm for the $τ$-quantile objective in finite-horizon Markov decision processes (MDPs). UCB-QRL is an iterative algorithm in which, at each iteration, we first estimate the underlying transition probability and then optimize the quantile value function over a confidence ball around this estimate. We show that UCB-QRL yields a high-probability regret bound $\mathcal O\left((2/κ)^{H+1}H\sqrt{SATH\log(2SATH/δ)}\right)$ in the episodic setting with $S$ states, $A$ actions, $T$ episodes, and $H$ horizons. Here, $κ>0$ is a problem-dependent constant that captures the sensitivity of the underlying MDP's quantile value.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。