提出新算法，首次实现熵风险下最优策略识别的最优样本复杂度。
Tight Sample Complexity Bounds for Entropic Best Policy Identification
- 基于前向模型与KL探索奖励，设计新型算法
- 样本复杂度达Ω(e^{|β|H})，匹配已知下界
- 适合关注风险敏感强化学习的科研人员
我们研究在熵风险度量下的有限时域风险敏感强化学习中的最优策略识别问题。已有研究表明，样本复杂度的上下界之间存在指数级时域依赖的恒定差距：已知下界为Ω(e^{|β| H})，而当前最优上界仅为O(e^{2|β| H})（arXiv:2506.00286v2），且基于生成模型。本文发现该额外指数因子源于对指数效用的过松集中控制。通过重新分析该问题，我们提出一种基于前向模型的算法，结合适配熵准则的KL探索奖励。改进主要来自两项关键技术：利用指数效用的光滑性推导更紧的集中界；设计新的停止规则，进一步利用此紧性，最终实现与下界一致的样本复杂度。
原文摘要 · Abstract (English)
We study best-policy identification for finite-horizon risk-sensitive reinforcement learning under the entropic risk measure. Recent work established a constant gap in the exponential horizon dependence between lower and upper bounds on the number of samples required to identify an approximately optimal policy. Precisely, known lower bounds scale in $Ω(e^{|β| H})$ where $H$ is the horizon of the MDP, while the state-of-the-art upper bound achieves at best $O(e^{2|β| H})$ (arXiv:2506.00286v2) using a generative model. We show that this extra exponential factor can be traced to overly loose concentration control for exponential utilities. To close this open gap, we revisit the analysis of this problem through a forward-model based algorithm building on KL-based exploration bonuses that we adapt to the entropic criterion. The improvement we get is due to two main novel technical innovations. We leverage the smoothness properties of the exponential utility to derive sharper concentration bounds, and we propose a new stopping rule that exploits further this tightness to obtain a sample complexity that matches the lower bound.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。