提出无需批量更新的增量式强化学习方法，让机器人在资源受限设备上也能高效训练。
Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers
- 采用增量更新机制，不依赖经验回放或批量处理
- 在小型回放缓冲区下仍能稳定训练，最终性能媲美批处理方法
- 首次实现真实机器人仅用增量更新完成深度强化学习
现代深度策略梯度方法在模拟机器人任务中表现优异，但均需大型经验回放缓冲区或昂贵的批量更新，难以应用于资源受限的真实系统。我们发现这些方法在小缓冲区或增量学习场景下会严重失效，即仅使用最新样本进行更新时。为此，提出一种新型增量式深度策略梯度方法——动作价值梯度（AVG），并结合归一化与缩放技术以解决增量学习中的不稳定性问题。在机器人仿真基准测试中，AVG是唯一能在增量模式下有效学习的方法，最终性能常接近批处理方法。该进展首次实现了仅使用增量更新的真实机器人深度强化学习，成功应用于机械臂和移动机器人。
原文摘要 · Abstract (English)
Modern deep policy gradient methods achieve effective performance on simulated robotic tasks, but they all require large replay buffers or expensive batch updates, or both, making them incompatible for real systems with resource-limited computers. We show that these methods fail catastrophically when limited to small replay buffers or during incremental learning, where updates only use the most recent sample without batch updates or a replay buffer. We propose a novel incremental deep policy gradient method -- Action Value Gradient (AVG) and a set of normalization and scaling techniques to address the challenges of instability in incremental learning. On robotic simulation benchmarks, we show that AVG is the only incremental method that learns effectively, often achieving final performance comparable to batch policy gradient methods. This advancement enabled us to show for the first time effective deep reinforcement learning with real robots using only incremental updates, employing a robotic manipulator and a mobile robot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。