提出近最优采样复杂度的迭代CVaR强化学习算法,兼顾风险控制与样本效率。
Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Model
- 基于价值迭代设计新算法ICVaR-VI,实现风险敏感策略优化
- 理论证明采样复杂度上界为$\tilde{O}(SA/(1-γ)^4τ^2ε^2)$,下界匹配
- 适用于高风险敏感场景,尤其适合小风险容忍度下的最坏路径规划
本文研究带生成模型的风险敏感强化学习的采样复杂度问题,目标是在每一步最大化条件风险价值(CVaR)且风险容忍度为$τ$,即迭代CVaR。我们建立了迭代CVaR RL与$(s,a)$-矩形分布鲁棒RL之间的联系,给出了该问题的近似上下界。证明基于值迭代的算法ICVaR-VI在$\tilde{O}(SA/(1-γ)^4τ^2ε^2)$样本内可达到$ε$-最优策略;当$τ\geq γ$时,采样复杂度优化至$\tilde{O}(SA/(1-γ)^3ε^2)$。进一步给出最小最大下界$\tilde{O}((1-γτ)SA/(1-γ)^4τε^2)$。对固定$τ\in(0,1]$,上下界一致,证明分析紧致性与最优性。此外,研究了小$τ$情形——最坏路径强化学习(Worst-Path RL),建立上下界均为$\tilde{O}(SA/p_{\min})$,其中$ p_{\min} $为转移核中最小非零可达概率。
原文摘要 · Abstract (English)
In this work, we study the sample complexity problem of risk-sensitive Reinforcement Learning (RL) with a generative model, where we aim to maximize the Conditional Value at Risk (CVaR) with risk tolerance level $τ$ at each step, a criterion we refer to as Iterated CVaR. We first build a connection between Iterated CVaR RL and $(s, a)$-rectangular distributional robust RL with a specific uncertainty set for CVaR. We establish nearly matching upper and lower bounds on the sample complexity of this problem. Specifically, we first prove that a value iteration-based algorithm, ICVaR-VI, achieves an $ε$-optimal policy with at most $\tilde{O} \left(\frac{SA}{(1-γ)^4τ^2ε^2} \right)$ samples, where $γ$ is the discount factor, and $S, A$ are the sizes of the state and action spaces. Furthermore, when $τ\geq γ$, the sample complexity improves to $\tilde{O} \left( \frac{SA}{(1-γ)^3ε^2} \right)$. We further show a minimax lower bound of $\tilde{O} \left(\frac{(1-γτ)SA}{(1-γ)^4τε^2} \right)$. For a fixed risk level $τ\in (0,1]$, our upper and lower bounds match, demonstrating the tightness and optimality of our analysis. We also investigate a limiting case with a small risk level $τ$, called Worst-Path RL, where the objective is to maximize the minimum possible cumulative reward. We develop matching upper and lower bounds of $\tilde{O} \left(\frac{SA}{p_{\min}} \right)$, where $p_{\min}$ denotes the minimum non-zero reaching probability of the transition kernel.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。