提出低开销强化学习算法，兼顾快速收敛与少切换次数。
Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learning
- 设计新算法，实现线性烧入成本和对数级策略切换
- 在单智能体和联邦场景下均达最优近似后悔值
- 适合数据采集昂贵的现实应用，如工业控制与分布式学习
针对单智能体和联邦强化学习中数据收集与策略部署成本高昂的问题，本文研究如何最小化烧入成本（达到近优后悔所需的样本量）以及策略切换或通信成本。在具有 $S$ 个状态和 $A$ 个动作的有限时域马尔可夫决策过程（MDPs）中，现有方法要么烧入成本超线性依赖于 $S$ 与 $A$，要么无法实现对数级切换或通信成本。本文提出两种新型无模型算法——Q-EarlySettled-LowCost 与 FedQ-EarlySettled-LowCost，首次同时实现：(i) 所有已知无模型强化学习或联邦强化学习算法中的最优近似后悔值；(ii) 线性依赖于 $S$ 与 $A$ 的低烧入成本；(iii) 单智能体下的对数级策略切换成本或联邦场景下的对数级通信成本。此外，我们建立了关于遗憾与切换/通信成本的依赖间隙的理论保证，其边界优于或匹配当前最优水平。
原文摘要 · Abstract (English)
Motivated by real-world settings where data collection and policy deployment -- whether for a single agent or across multiple agents -- are costly, we study the problem of on-policy single-agent reinforcement learning (RL) and federated RL (FRL) with a focus on minimizing burn-in costs (the sample sizes needed to reach near-optimal regret) and policy switching or communication costs. In parallel finite-horizon episodic Markov Decision Processes (MDPs) with $S$ states and $A$ actions, existing methods either require superlinear burn-in costs in $S$ and $A$ or fail to achieve logarithmic switching or communication costs. We propose two novel model-free RL algorithms -- Q-EarlySettled-LowCost and FedQ-EarlySettled-LowCost -- that are the first in the literature to simultaneously achieve: (i) the best near-optimal regret among all known model-free RL or FRL algorithms, (ii) low burn-in cost that scales linearly with $S$ and $A$, and (iii) logarithmic policy switching cost for single-agent RL or communication cost for FRL. Additionally, we establish gap-dependent theoretical guarantees for both regret and switching/communication costs, improving or matching the best-known gap-dependent bounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。