用少量在线数据高效优化离线强化学习，显著提升学习效率。
Sample Efficient Active Algorithms for Offline Reinforcement Learning
- 基于高斯过程建模不确定性，主动选择关键数据进行在线交互
- 理论证明只需约 1/ε² 次在线交互即可达到 ε-最优策略
- 适合追求低样本开销的离线强化学习研究者
离线强化学习（Offline RL）依赖静态数据训练策略，常面临状态-动作空间覆盖不足和分布偏移问题。通过引入有限在线交互，主动精炼价值函数不确定区域，可缓解该问题，称为主动强化学习（ActiveRL）。尽管已有良好实证表现，但缺乏理论分析。本文从高斯过程（GP）不确定性建模视角，首次提供严格的样本复杂度分析。提出一种算法，结合GP集中不等式与信息增益边界，推导出高概率保证：仅需 𝒪(1/ε²) 次主动转移即可获得 ε-最优策略，优于纯离线方法 Ω(1/ε²(1−γ)⁴) 的速率。结果表明，ActiveRL具备近似最优的信息效率，即通过引导不确定性减少，实现价值函数快速收敛且仅需极少在线数据。理论框架融合贝叶斯非参数回归与强化学习，实验验证了算法与理论结论的有效性。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) enables policy learning from static data but often suffers from poor coverage of the state-action space and distributional shift problems. This problem can be addressed by allowing limited online interactions to selectively refine uncertain regions of the learned value function, which is referred to as Active Reinforcement Learning (ActiveRL). While there has been good empirical success, no theoretical analysis is available in the literature. We fill this gap by developing a rigorous sample-complexity analysis of ActiveRL through the lens of Gaussian Process (GP) uncertainty modeling. In this respect, we propose an algorithm and using GP concentration inequalities and information-gain bounds, we derive high-probability guarantees showing that an $ε$-optimal policy can be learned with ${\mathcal{O}}(1/ε^2)$ active transitions, improving upon the $Ω(1/ε^2(1-γ)^4)$ rate of purely offline methods. Our results reveal that ActiveRL achieves near-optimal information efficiency, that is, guided uncertainty reduction leads to accelerated value-function convergence with minimal online data. Our analysis builds on GP concentration inequalities and information-gain bounds, bridging Bayesian nonparametric regression and reinforcement learning theories. We conduct several experiments to validate the algorithm and theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。