提出新算法，让重尾马尔可夫决策在对抗与随机环境下都表现优异。
Best-of-Both-Worlds for Heavy-Tailed Markov Decision Processes
- 基于经验分布与跳过损失估计，设计双模式优化框架。
- 对抗环境下降落至 $\widetilde{O}(T^{1/α})$，随机环境为 $O(\log T)$。
- 适用于有重尾损失的强化学习场景，尤其适合鲁棒性要求高的应用。
我们研究具有重尾损失（HTMDPs）的周期性马尔可夫决策过程。现有方法在随机环境中过于保守，在对抗性环境下缺乏自适应能力。本文提出两种算法：HT-FTRL-OM 和 HT-FTRL-UOB，实现‘双优’（BoBW）性能：在对抗环境中获得与实例无关的后悔值，在自约束（包括随机情形）环境中达到对数级实例相关后悔值。对于已知转移情况，HT-FTRL-OM 在占用量度上应用跟随正则化领导者（FTRL）框架，并引入新型跳过损失估计器，实现对抗环境下 $\widetilde{O}(T^{1/α})$ 的后悔界，随机环境下 $O(\log T)$。在此基础上，针对更困难的未知转移情形，提出新算法 HT-FTRL-UOB，假设损失分布满足温和截断非负条件，采用悲观跳过损失估计器，在对抗环境下达到 $\widetilde{O}(T^{1/α} + \sqrt{T})$，随机环境下 $O(\log^2 T)$。分析通过局部控制重尾偏移损失、新次优质量传播原理及分离转移不确定性与重尾估计误差和跳过偏差的后悔分解等技术突破关键障碍。
原文摘要 · Abstract (English)
We investigate episodic Markov Decision Processes with heavy-tailed losses (HTMDPs). Existing approaches for HTMDPs are conservative in stochastic environments and lack adaptivity in adversarial regimes. In this work, we propose algorithms HT-FTRL-OM and HT-FTRL-UOB for HTMDPs that achieve Best-of-Both-Worlds (BoBW) guarantees: instance-independent regret in adversarial environments and logarithmic instance-dependent regret in self-bounding (including the stochastic case) environments. For the known transition setting, HT-FTRL-OM applies the Follow-The-Regularized-Leader (FTRL) framework over occupancy measures with novel skipping loss estimators, achieving a $\widetilde{O}(T^{1/α})$ regret bound in adversarial regimes and a ${O}(\log T)$ regret in stochastic regimes. Building upon this framework, we develop a novel algorithm HT-FTRL-UOB to tackle the more challenging unknown-transition setting. Under a mild truncative nonnegativity condition on the loss distributions, this algorithm employs a pessimistic skipping loss estimator and achieves a $\widetilde{O}(T^{1/α} + \sqrt{T})$ regret in adversarial regimes and a ${O}(\log^2(T))$ regret in stochastic regimes. Our analysis overcomes key barriers through several technical insights, including a local control mechanism for heavy-tailed shifted losses, a new suboptimal-mass propagation principle, and a novel regret decomposition that isolates transition uncertainty from heavy-tailed estimation errors and skipping bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。