arXiv:2602.02260cs.LG2026-02

在完全不观察状态动作对的情况下,实现近似最优的强化学习性能。

Learning Markov Decision Processes under Fully Bandit Feedback

  • 设计首个高效算法,在仅知累计奖励的严格限制下学习马尔可夫决策过程。
  • 达到 $ ilde{O}( oot{T})$ 的后悔率,且对时长 $H$ 存在指数依赖性,此为必要。
  • 适用于 $k$-项先知不等式等经典优化问题,实测表现接近带完整反馈的先进算法。

强化学习中通常假设智能体能观测到每个访问的状态-动作对及每步奖励,已有理论证明在此设定下可实现近乎紧致的 $Θ( oot{T})$ 前悔界。然而,这种详细反馈可能不现实,近期研究探索了轨迹反馈(仅可观测全部状态-动作对,但仅得单一聚合奖励)等更受限情形。本文考虑一种更严格的“完全棋盘”反馈模型:智能体既不观察状态-动作对,也只知晓聚合奖励。我们提出首个针对回合制马尔可夫决策过程的高效棋盘学习算法,实现 $ ilde{O}( oot{T})$ 前悔。该后悔界对时长 $H$ 具有指数依赖性,我们证明其不可避免。此外,我们对“有序”MDP获得改进的近乎紧致后悔界,可用于建模 $k$-项先知不等式与顺序定价等经典随机优化问题。最后,我们在 $k$-项先知不等式场景中评估算法表现;尽管反馈极度受限,其性能仍与具备完整状态-动作反馈的先进算法(UCB-VI)相当。

原文摘要 · Abstract (English)

A standard assumption in Reinforcement Learning is that the agent observes every visited state-action pair in the associated Markov Decision Process (MDP), along with the per-step rewards. Strong theoretical results are known in this setting, achieving nearly-tight $Θ(\sqrt{T})$-regret bounds. However, such detailed feedback can be unrealistic, and recent research has investigated more restricted settings such as trajectory feedback, where the agent observes all the visited state-action pairs, but only a single \emph{aggregate} reward. In this paper, we consider a far more restrictive ``fully bandit'' feedback model for episodic MDPs, where the agent does not even observe the visited state-action pairs -- it only learns the aggregate reward. We provide the first efficient bandit learning algorithm for episodic MDPs with $\widetilde{O}(\sqrt{T})$ regret. Our regret has an exponential dependence on the horizon length $\H$, which we show is necessary. We also obtain improved nearly-tight regret bounds for ``ordered'' MDPs; these can be used to model classical stochastic optimization problems such as $k$-item prophet inequality and sequential posted pricing. Finally, we evaluate the empirical performance of our algorithm for the setting of $k$-item prophet inequalities; despite the highly restricted feedback, our algorithm's performance is comparable to that of a state-of-art learning algorithm (UCB-VI) with detailed state-action feedback.

强化学习在线学习后悔分析最优停止

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。