arXiv:2603.09950cs.LGcs.AI2026-03被引 2

用神经元激活模式提前判断PPO训练好坏,省下大量调参时间。

When Learning Rates Go Wrong: Early Structural Signals in PPO Actor-Critic

  • 通过激活符号变化量度(OUI)监测网络结构演化
  • 训练10%时就能区分学习率优劣,最高回报对应特定OUI区间
  • 比返利、翻转等早期指标更准,适合快速筛掉失败实验

深度强化学习对学习率极为敏感,稳定高效训练常需大量超参数调优。在近端策略优化(PPO)的演员-评论家方法中,过小的学习率导致收敛缓慢,过大则引发不稳定性或崩溃。本文通过分析网络隐藏神经元行为,使用过拟合-欠拟合指示器(OUI)量化固定探测批次上的二值激活模式平衡。提出高效的基于批次的OUI计算方式,并建立学习率与神经元激活符号变化之间的理论关联,揭示了正确内部结构演化依赖于步长。在三个离散控制环境及多个随机种子下,实验表明:仅训练10%时,OUI即可有效区分不同学习率范式。观察到一致的不对称性:获得最高回报的评论家网络处于中间OUI区间(避免饱和),而演员网络则表现出较高的OUI值。将OUI筛选规则与早期奖励、剪裁、发散、翻转等标准在相同成功运行召回率下对比,结果显示:OUI提供最强早期信号——单独使用即能在更广召回率下达到最佳精度;结合早期奖励后,可实现最高效筛选,在无需完整训练的前提下大幅剔除无效实验。

原文摘要 · Abstract (English)

Deep Reinforcement Learning systems are highly sensitive to the learning rate (LR), and selecting stable and performant training runs often requires extensive hyperparameter search. In Proximal Policy Optimization (PPO) actor--critic methods, small LR values lead to slow convergence, whereas large LR values may induce instability or collapse. We analyse this phenomenon from the behavior of the hidden neurons in the network using the Overfitting-Underfitting Indicator (OUI), a metric that quantifies the balance of binary activation patterns over a fixed probe batch. We introduce an efficient batch-based formulation of OUI and derive a theoretical connection between LR and activation sign changes, clarifying how a correct evolution of the neuron's inner structure depends on the step size. Empirically, across three discrete-control environments and multiple seeds, we show that OUI measured at only 10\% of training already discriminates between LR regimes. We observe a consistent asymmetry: critic networks achieving highest return operate in an intermediate OUI band (avoiding saturation), whereas actor networks achieving highest return exhibit comparatively high OUI values. We then compare OUI-based screening rules against early return, clip-based, divergence-based, and flip-based criteria under matched recall over successful runs. In this setting, OUI provides the strongest early screening signal: OUI alone achieves the best precision at broader recall, while combining early return with OUI yields the highest precision in best-performing screening regimes, enabling aggressive pruning of unpromising runs without requiring full training.

强化学习PPO早筛超参优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。