提出新方法,让弱耦合强化学习的样本复杂度从指数降为多项式。
Lyapunov-Based Sample Complexity Analysis for Weakly-Coupled MDPs
- 用李雅普诺夫函数构建新分析框架,避免传统偏差控制难题。
- 在弱耦合MDP中实现首次多项式复杂度的有限样本保证,最优差距为1/√N。
- 适用于多臂老虎机等弱耦合系统,适合研究高效强化学习的学者。
我们研究在生成模型下平均奖励弱耦合马尔可夫决策过程(WCMDPs)和非平稳多臂老虎机(RBs)的学习样本复杂度。直接转化为表格型MDP会导致状态-动作空间随臂数$N$指数增长,复杂度过高。通过利用弱耦合结构,我们证明近优策略可在多项式于$N$的样本与计算复杂度下学习到。具体地,分析了插件方法:对数据估计的经验模型应用高效规划算法。对于完全异构的WCMDPs,首次建立了多项式复杂度的有限样本概率保证,最优差距为$O(1/\\/sqrt{N})$。对于同质的RBs,进一步在温和结构假设下证明更小的最优差距可达。本工作的核心技术贡献是新的基于李雅普诺夫的分析框架。不同于依赖难控偏差函数的经典方法,本框架显式构造李雅普诺夫函数,并结合真实与经验模型间的漂移传递技术。框架中一个独立有意义的步骤是对底层线性规划松弛的精细扰动分析,为分析基于线性规划的策略及弱耦合系统提供通用工具。
原文摘要 · Abstract (English)
We study the sample complexity of learning in average-reward weakly-coupled Markov decision processes (WCMDPs) and Restless Bandits (RBs) under a generative model. Naive reduction to a tabular MDP leads to high complexity bounds as the state-action space is exponentially large in the number of arms $N$. By exploiting the weakly coupled structure, we show that near-optimal policies can be learned with sample and computational complexities that are polynomial in $N$. Specifically, we analyze the plug-in approach, which applies an efficient planning algorithm to an empirical model estimated from data. For fully heterogeneous WCMDPs, we establish the first finite-sample PAC guarantee with polynomial complexity and an $O(1/\sqrt{N})$ optimality gap. For homogeneous RBs, we further prove that a smaller optimality gap is achievable under mild structural assumptions. A primary technical contribution of our work is a novel Lyapunov-based analysis framework. Unlike classical approaches that rely on the difficult-to-control bias function, our framework uses an explicitly constructed Lyapunov function along with a drift transfer technique between the true and empirical models. A key step of independent interest in our framework is a fine-grained perturbation analysis for the underlying linear programming (LP) relaxation, which provides a general tool for analyzing LP-based policies and weakly-coupled systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。