arXiv:2505.17714cs.LGcs.AI2025-05

PPO-BR动态调节信任区域,让强化学习更快收敛且更稳定。

PPO-BR: Dual-Signal Entropy-Reward Adaptation for Trust Region Policy Optimization

  • 根据熵和奖励信号动态调整更新范围,兼顾探索与收敛。
  • 在六项基准上实现29.1%更快收敛,奖励方差降低2.3倍。
  • 代码改动少于5行,适合医疗机器人等高安全场景部署。

尽管近端策略优化(PPO)在从机器人控制到游戏AI的领域中占据主导地位,其固定的信任区域导致脆弱的权衡:激进裁剪抑制早期探索,后期更新则破坏收敛稳定性。PPO-BR通过融合探索与收敛信号,建立自适应强化学习的新范式——一个理论严谨的创新,以低于2%的开销超越五种最先进基线。该方法在六个多样化基准(MuJoCo、Atari、稀疏奖励环境)上实现了29.1%的更快收敛(p < 0.001),奖励方差降低2.3倍,运行时开销小于1.8%,仅需五行代码修改。其简洁性与理论保证使其可直接应用于医疗机器人、自动驾驶无人机等安全关键系统。相比近期方法如组相对策略优化(GRPO),PPO-BR提供统一的熵-奖励机制,适用于语言模型与通用强化学习环境。

原文摘要 · Abstract (English)

Despite Proximal Policy Optimization (PPO) dominating policy gradient methods -- from robotic control to game AI -- its static trust region forces a brittle trade-off: aggressive clipping stifles early exploration, while late-stage updates destabilize convergence. PPO-BR establishes a new paradigm in adaptive RL by fusing exploration and convergence signals into a single bounded trust region -- a theoretically grounded innovation that outperforms five SOTA baselines with less than 2% overhead. This work bridges a critical gap in phase-aware learning, enabling real-world deployment in safety-critical systems like robotic surgery within a single adaptive mechanism. PPO-BR achieves 29.1% faster convergence by combining: (1) entropy-driven expansion (epsilon up) for exploration in high-uncertainty states, and (2) reward-guided contraction (epsilon down) for convergence stability. On six diverse benchmarks (MuJoCo, Atari, sparse-reward), PPO-BR achieves 29.1% faster convergence (p < 0.001), 2.3x lower reward variance than PPO, and less than 1.8% runtime overhead with only five lines of code change. PPO-BR's simplicity and theoretical guarantees make it ready-to-deploy in safety-critical domains -- from surgical robotics to autonomous drones. In contrast to recent methods such as Group Relative Policy Optimization (GRPO), PPO-BR offers a unified entropy-reward mechanism applicable to both language models and general reinforcement learning environments.

强化学习策略优化自适应控制机器人

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。