提出约束型可训练激活函数,平衡强化学习中的表达能力与稳定性。
Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning
- 用有理函数构建可训练激活,通过结构约束控制输出膨胀。
- 在连续控制任务中显著提升训练稳定性和最终性能,减少过估计问题。
- 适用于需要长期适应的强化学习与持续学习场景,尤其适合连续动作空间。
可训练激活函数(参数随网络权重优化)相比固定激活函数具有更强的表达能力。特别是以多项式比形式定义的有理函数激活,在强化学习中提升了模型的可塑性。然而,其对训练稳定性的影响尚不明确。本文研究了有理激活在强化学习与持续学习中的表现,发现其灵活性虽增强适应性,但也引发不稳定,导致强化学习中出现过估计,持续学习中发生特征坍缩。核心发现是:有理激活存在表达能力与可塑性之间的权衡。为此,我们提出一种结构化约束的变体,限制输出过度放大,同时保持适应能力。在MetaWorld和DeepMind Control Suite(DMC)环境中的实验表明,该方法提升了训练稳定性和性能。在持续学习基准测试(包括标签重排的MNIST和拆分的CIFAR-100)中,揭示了不同约束对表达能力与长期记忆的平衡影响。初步在离散动作空间(如Atari)未观察到类似不稳定性,提示该权衡在连续控制中尤为关键。研究为动态非平稳环境中的鲁棒可训练激活设计提供了实用原则。代码见:https://github.com/special114/rl_rational_plasticity。
原文摘要 · Abstract (English)
Trainable activation functions, whose parameters are optimized alongside network weights, offer increased expressivity compared to fixed activation functions. Specifically, trainable activation functions defined as ratios of polynomials (rational functions) have been proposed to enhance plasticity in reinforcement learning. However, their impact on training stability remains unclear. In this work, we study trainable rational activations in both reinforcement and continual learning settings. We find that while their flexibility enhances adaptability, it can also introduce instability, leading to overestimation in RL and feature collapse in longer continual learning scenarios. Our main result is demonstrating a trade-off between expressivity and plasticity in rational activations. To address this, we propose a constrained variant that structurally limits excessive output scaling while preserving adaptability. Experiments across MetaWorld and DeepMind Control Suite (DMC) environments show that our approach improves training stability and performance. In continual learning benchmarks, including MNIST with reshuffled labels and Split CIFAR-100, we reveal how different constraints affect the balance between expressivity and long-term retention. While preliminary experiments in discrete action domains (e.g., Atari) did not show similar instability, this suggests that the trade-off is particularly relevant for continuous control. Together, our findings provide actionable design principles for robust and adaptable trainable activations in dynamic, non-stationary environments. Code available at: https://github.com/special114/rl_rational_plasticity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。