解决无人机控制中奖励函数部分可导导致的梯度偏差问题
ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards
- 用0步与N步回报结合，利用学习到的Q值梯度减少偏差
- 在真实和仿真环境下均实现更快收敛和更高最终奖励
- 适合需要高效强化学习的机器人控制场景
四旋翼飞行器控制策略可通过精确奖励梯度，直接使用反向传播通过时间（BPTT）优化策略参数获得高性能。然而，设计完全可导的奖励架构往往困难，部分可导奖励会导致梯度传播偏差，降低训练性能。为此，我们提出修正的反向传播通过时间（ABPT），在保持BPTT训练效率的同时缓解梯度偏差。ABPT结合0步与N步回报，有效利用学习到的Q值函数梯度来减小偏差，并采用熵正则化与状态初始化机制，提升训练过程中的探索能力。我们在四个典型四旋翼飞行任务上进行评估，涵盖真实世界与仿真环境。实验结果表明，相较于现有算法，ABPT收敛速度显著更快，最终奖励更高，尤其在部分可导奖励任务中表现突出。代码将发布于 http://github.com/Fanxing-LI/ABPT。
原文摘要 · Abstract (English)
Quadrotor control policies can be trained with high performance using the exact gradients of the rewards to directly optimize policy parameters via backpropagation-through-time (BPTT). However, designing a fully differentiable reward architecture is often challenging. Partially differentiable rewards will result in biased gradient propagation that degrades training performance. To overcome this limitation, we propose Amended Backpropagation-through-Time (ABPT), a novel approach that mitigates gradient bias while preserving the training efficiency of BPTT. ABPT combines 0-step and N-step returns, effectively reducing the bias by leveraging value gradients from the learned Q-value function. Additionally, it adopts entropy regularization and state initialization mechanisms to encourage exploration during training. We evaluate ABPT on four representative quadrotor flight tasks \li{in both real world and simulation}. Experimental results demonstrate that ABPT converges significantly faster and achieves higher ultimate rewards than existing learning algorithms, particularly in tasks involving partially differentiable rewards. The code will be released at http://github.com/Fanxing-LI/ABPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。