给奖励加噪声，让智能体探索更全面，提升强化学习效果。
Exploration by Random Reward Perturbation
- 向环境奖励添加零均值噪声，增强策略多样性。
- 在稀疏和密集奖励下均显著提升采样效率，避免局部最优。
- 轻量易用，可无缝集成到现有算法中，适合多数强化学习场景。
我们提出随机奖励扰动（RRP），一种新型强化学习探索策略。理论分析表明，在环境奖励上添加零均值噪声能有效提升训练过程中的策略多样性，扩大探索范围。RRP与基于动作扰动的探索策略（如ε-贪婪、随机策略、熵正则化）完全兼容，可带来叠加性探索提升。该方法通用、轻量，可低开销集成到现有RL算法中。RRP建立了奖励塑造与噪声驱动探索之间的理论联系，揭示二者互补潜力。实验显示，RRP显著提升近端策略优化（PPO）和软演员-评论家（SAC）的性能，在多种任务中实现更高样本效率，并在稀疏与密集奖励场景下均能逃离局部最优。
原文摘要 · Abstract (English)
We introduce Random Reward Perturbation (RRP), a novel exploration strategy for reinforcement learning (RL). Our theoretical analyses demonstrate that adding zero-mean noise to environmental rewards effectively enhances policy diversity during training, thereby expanding the range of exploration. RRP is fully compatible with the action-perturbation-based exploration strategies, such as $ε$-greedy, stochastic policies, and entropy regularization, providing additive improvements to exploration effects. It is general, lightweight, and can be integrated into existing RL algorithms with minimal implementation effort and negligible computational overhead. RRP establishes a theoretical connection between reward shaping and noise-driven exploration, highlighting their complementary potential. Experiments show that RRP significantly boosts the performance of Proximal Policy Optimization and Soft Actor-Critic, achieving higher sample efficiency and escaping local optima across various tasks, under both sparse and dense reward scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。