arXiv:2603.01309cs.LGstat.ML2026-03

为强化学习提供可信赖的样本效率保证,让算法在数据少时仍能安全可用。

PAC Guarantees for Reinforcement Learning: Sample Complexity, Coverage, and Structure

  • 提出覆盖-结构-目标框架,拆解强化学习理论中的关键因素
  • 揭示不同模型(线性、核函数等)在有限数据下的学习边界
  • 适合关注算法可靠性与理论保障的研究者和工程师

当数据稀缺或错误代价高昂时,平均性能指标难以满足需求。从业者真正需要的是概率保证:以至少 $1-δ$ 的置信度,在 $N$ 个训练周期后,学到的策略与最优策略差距不超过 $\varepsilon$。这就是 PAC 保证。2018 至 2025 年间,强化学习理论领域在实现此类保证方面取得显著进展。本文综述这一进展,引入覆盖-结构-目标(CSO)框架,将几乎所有的 PAC 样本复杂度结果分解为三部分:覆盖(数据获取方式)、结构(马尔可夫决策过程或函数类的内在复杂度)、目标(学习者需达成的性能)。该框架虽非定理,但作为解释工具,可识别瓶颈并实现跨场景对比。核心技术涵盖紧致的表格型基线、统一 PAC 与后悔率的桥梁;控制可学习性的结构复杂度度量(如 Bellman 秩、见证秩、贝尔曼-尤勒维度);线性、核/NTK 和低秩模型的结果;奖励无关探索作为前期覆盖投入;以及悲观离线强化学习中继承覆盖为关键约束。本文还提供实用工具:按 CSO 坐标索引的速率查表、贝尔曼残差诊断、部署门控的覆盖率估计、每轮策略证书。最后一节列出开放问题,区分近期目标与当前理论尚无法解决的前沿难题。

原文摘要 · Abstract (English)

When data is scarce or mistakes are costly, average-case metrics fall short. What a practitioner needs is a guarantee: with probability at least $1-δ$, the learned policy is $\varepsilon$-close to optimal after $N$ episodes. This is the PAC promise, and between 2018 and 2025 the RL theory community made striking progress on when such promises can be kept. We survey that progress. Our organizing tool is the Coverage-Structure-Objective (CSO) framework, proposed here, which decomposes nearly every PAC sample complexity result into three factors: coverage (how data were obtained), structure (intrinsic MDP or function-class complexity), and objective (what the learner must deliver). CSO is not a theorem but an interpretive template that identifies bottlenecks and makes cross-setting comparison immediate. The technical core covers tight tabular baselines and the uniform-PAC bridge to regret; structural complexity measures (Bellman rank, witness rank, Bellman-Eluder dimension) governing learnability with function approximation; results for linear, kernel/NTK, and low-rank models; reward-free exploration as upfront coverage investment; and pessimistic offline RL where inherited coverage is the binding constraint. We provide practitioner tools: rate lookup tables indexed by CSO coordinates, Bellman residual diagnostics, coverage estimation with deployment gates, and per-episode policy certificates. A final section catalogs open problems, separating near-term targets from frontier questions where coverage, structure, and computation tangle in ways current theory cannot resolve.

强化学习理论保证样本效率算法安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。