提出局部重置机制，防止持续强化学习中策略崩溃。
Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning
- 按神经元有用性加权部分重置，避免全量重置损伤性能。
- 在4亿步训练中唯一避免策略崩溃，优于现有重置与衰减方法。
- 适合长期持续学习场景，可调节可塑性与最优性能平衡。
神经网络在非平稳数据环境中（如持续监督与强化学习）会因冗余神经元累积和表达能力下降而性能衰退。近期通过神经元重置可恢复梯度流动与可塑性，但全单元重初始化常导致峰值性能下降并引发训练不稳定，造成策略崩溃。为此，本文提出校准的局部重置（CPR）优化器，定期将低效神经元向其初始值拉回，拉力大小由神经元实用性决定。相比二值重置，部分重置更具鲁棒性；相比均匀衰减，校准的实用性加权使调整聚焦于最需要的单元。在连续学习基准测试中，仅CPR在4亿步训练内未出现策略崩溃，且在Continual MetaWorld与Continual MinAtar任务上优于以往基于衰减与重置的方法。消融实验揭示了可调的可塑性与峰值性能权衡，验证了实用性加权重初始化在持续学习中的潜力。
原文摘要 · Abstract (English)
Neural networks are hindered by accumulating dormant neurons and loss of expressivity throughout training, particularly in non-stationary data settings, such as continual supervised and reinforcement learning. Recently, neuron resets have been used to maintain gradient flow and restore plasticity. However, full unit reinitialization often sacrifices peak performance and can destabilize training, leading to policy collapse. To preserve plasticity without destabilizing training, we propose Calibrated Partial Resets (CPR), an optimizer that periodically pulls low-utility neurons toward their initialization, with pull strength scaled by each neuron's utility. Unlike binary reset methods, partial resets avoid brittleness; unlike uniform decay, calibrated utility-scaling concentrates adjustment on the units that need it most. Among compared methods, only CPR avoids policy collapse over 400M training steps in SlipperyAnt, and it outperforms prior decay and reset-based methods on Continual MetaWorld and Continual MinAtar benchmarks. Ablations reveal a tunable trade-off between plasticity and peak performance, highlighting utility-scaled reinitialization as a promising direction for continual learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。