解决强化学习中凸目标函数的在线学习问题，首次实现近最优后悔界。
Online Episodic Convex Reinforcement Learning
- 基于在线镜像下降与动态约束集设计新算法
- 在无转移函数先验下达到近最优后悔率
- 首次处理仅反馈目标值的带通货币版本，适合数据稀缺场景
我们研究具有凸目标函数的回合制有限时域马尔可夫决策过程（MDP）中的在线学习问题，即凹效用强化学习（CURL）问题。该设定将强化学习从线性损失推广至状态-动作分布上的凸损失，非线性导致经典贝尔曼方程失效，需新算法。本文提出首个无需任何转移函数先验即可实现近最优后悔界的方法，采用在线镜像下降配合动态约束集与精心设计的探索奖励。进一步首次解决了带通货货币版本的CURL问题，仅能获取策略诱导的状态-动作分布上的目标函数值。通过将带通货货币凸优化技术适配到MDP框架，实现了次线性后悔界。
原文摘要 · Abstract (English)
We study online learning in episodic finite-horizon Markov decision processes (MDPs) with convex objective functions, known as the concave utility reinforcement learning (CURL) problem. This setting generalizes RL from linear to convex losses on the state-action distribution induced by the agent's policy. The non-linearity of CURL invalidates classical Bellman equations and requires new algorithmic approaches. We introduce the first algorithm achieving near-optimal regret bounds for online CURL without any prior knowledge on the transition function. To achieve this, we use an online mirror descent algorithm with varying constraint sets and a carefully designed exploration bonus. We then address for the first time a bandit version of CURL, where the only feedback is the value of the objective function on the state-action distribution induced by the agent's policy. We achieve a sub-linear regret bound for this more challenging problem by adapting techniques from bandit convex optimization to the MDP setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。