arXiv:2607.04990cs.LG2026-07

用非凸正则化提升强化学习特征选择,解决传统方法偏差问题。

Non-Convex Sparse Reinforcement Learning via Non-Monotone Inclusions

  • 引入非凸投影极小极大凹惩罚,改进经典LSTD估计。
  • 在噪声特征多的场景下,性能显著优于现有方法。
  • 理论贡献:拓展FRBS算法收敛性分析至更广非单调包含问题。

本文有两个关键贡献:一是高效强化学习中的特征选择方法,二是非单调包含理论的进展。在强化学习方面,通过在经典最小二乘时间差(LSTD)策略评估中加入诱导稀疏性的非凸投影极小极大凹(PMC)惩罚,解决了传统正则化方案固有的估计偏差问题。由于PMC惩罚是弱凸的,导致的不动点问题不再单调,而是属于更广泛的非单调包含类问题,即单调Lipschitz算子与拟单调算子之和。理论上,针对该类问题,发展了前向-反射-后向分裂(FRBS)方法的新收敛条件:在较弱条件下,建立迭代序列的李雅普诺夫稳定性及极限点存在性;在弱Minty变分不等式假设下,可保证精确收敛。基准数据集上的数值实验表明,将所提FRBS迭代应用于非凸正则化LSTD问题时,在存在大量噪声特征的情况下,显著优于当前最先进的特征选择方法。

原文摘要 · Abstract (English)

This work delivers two key contributions: one to efficient feature selection in reinforcement learning (RL), the other to the theory of non-monotone inclusions. On the RL side, the estimation bias inherent in conventional regularization schemes is addressed by augmenting classical least-squares temporal-difference (LSTD) policy evaluation with the sparsity-inducing, non-convex projected minimax concave (PMC) penalty. Because the PMC penalty is weakly convex, the resulting fixed-point problem is no longer monotone; instead, it falls under a broader class of non-monotone inclusions involving the sum of a monotone Lipschitz operator and a hypomonotone operator. On the theory side, novel convergence conditions are developed for the forward-reflected-backward splitting (FRBS) method applied to this broader class of non-monotone inclusion problems. Under mild conditions, Lyapunov stability and the existence of a limit point of the sequence of FRBS iterates are established; alternatively, under the weak Minty variational inequality assumption, exact convergence is guaranteed. Numerical tests on benchmark datasets show that the proposed FRBS iterates, applied to the non-convexly regularized LSTD problem, substantially outperform state-of-the-art feature-selection methods, especially when many noisy features are present.

强化学习稀疏性非凸优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。