提出递归熵风险学习算法，首次给出采样复杂度的严格理论保证。
Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Model
- 基于生成模型设计模型化算法MB-RS-QVI，处理风险敏感强化学习
- 采样复杂度随|β|/(1-γ)指数增长，且该依赖关系不可避免
- 适用于需权衡风险偏好（保守或激进）的决策场景
研究在有限折扣MDP中使用递归熵风险度量（ERM）的风险敏感强化学习，其中风险参数β≠0控制智能体的风险态度：β>0为规避风险，β<0为追求风险。假设存在MDP的生成模型。重点分析在递归ERM下学习最优状态-动作值函数（值学习）和最优策略（策略学习）的采样复杂度。提出一种基于模型的算法——模型化递归风险敏感Q值迭代（MB-RS-QVI），并推导出其在值学习与策略学习上的概率近似正确（PAC）型采样复杂度界。两个界的复杂度均随|β|/(1-γ)指数级增长，其中γ为折扣因子。同时建立相应的下界，表明在最坏情况下，这种对|β|/(1-γ)的指数依赖是不可避免的。界在状态数S和动作数A上是紧的，首次为递归ERM在风险规避与风险追求两种情形下提供了严格的采样复杂度保障。
原文摘要 · Abstract (English)
We study risk-sensitive reinforcement learning in finite discounted MDPs with recursive entropic risk measures (ERM), where the risk parameter $β\neq 0$ controls the agent's risk attitude: $β>0$ for risk-averse and $β<0$ for risk-seeking behavior. A generative model of the MDP is assumed to be available. Our focus is on the sample complexities of learning the optimal state-action value function (value learning) and an optimal policy (policy learning) under recursive ERM. We introduce a model-based algorithm, called Model-Based ERM $Q$-Value Iteration (MB-RS-QVI), and derive PAC-type bounds on its sample complexity for both value and policy learning. Both PAC bounds scale exponentially with $|β|/(1-γ)$, where $γ$ is the discount factor. We also establish corresponding lower bounds for both value and policy learning, showing that exponential dependence on $|β|/(1-γ)$ is unavoidable in the worst case. The bounds are tight in the number of states and actions ($S$ and $A$), providing the first rigorous sample complexity guarantees for recursive ERM across both risk-averse and risk-seeking regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。