通过渐进衰减基策略,让机器人赛车更高效稳定。
Efficient Real-World Autonomous Racing via Attenuated Residual Policy Optimization
- 用衰减机制将基策略融入新策略,降低部署复杂度。
- 实测在仿真和真实赛车中均优于基线,零样本迁移成功。
- 适合需要低延迟、高可靠性的自主控制场景。
残差策略学习(RPL)通过深度强化学习微调静态基策略,在多种机器人应用中表现优异,尤其在自动驾驶赛车这一极具挑战性的领域。然而,基于RPL的控制器部署会引入系统复杂性并增加推理延迟。为此,本文提出一种RPL扩展方法——衰减残差策略优化(α-RPO)。与标准RPL不同,α-RPO通过逐步衰减基策略的影响,生成一个独立的神经策略,初始阶段基策略用于引导学习。该机制还支持特权学习,允许基策略使用最终部署时无需的传感器模态。α-RPO与PPO无缝集成,动态补偿基控制器的影响。我们在1:10比例的自动驾驶赛车框架中评估了该方法。在仿真及零样本真实世界迁移至Roboracer车辆中,α-RPO不仅降低了系统复杂度,还提升了驾驶性能,验证了其在机器人部署中的实用性。代码已开源:https://github.com/raphajaner/arpo_racing。
原文摘要 · Abstract (English)
Residual policy learning (RPL), in which a learned policy refines a static base policy using deep reinforcement learning (DRL), has shown strong performance across various robotic applications. Its effectiveness is particularly evident in autonomous racing, a domain that serves as a challenging benchmark for real-world DRL. However, deploying RPL-based controllers introduces system complexity and increases inference latency. We address this by introducing an extension of RPL named attenuated residual policy optimization ($α$-RPO). Unlike standard RPL, $α$-RPO yields a standalone neural policy by progressively attenuating the base policy, which initially serves to bootstrap learning. Furthermore, this mechanism enables a form of privileged learning, where the base policy is permitted to use sensor modalities not required for final deployment. We design $α$-RPO to integrate seamlessly with PPO, ensuring that the attenuated influence of the base controller is dynamically compensated during policy optimization. We evaluate $α$-RPO by building a framework for 1:10-scaled autonomous racing around it. In both simulation and zero-shot real-world transfer to Roboracer cars, $α$-RPO not only reduces system complexity but also improves driving performance compared to baselines - demonstrating its practicality for robotic deployment. Our code is available at: https://github.com/raphajaner/arpo_racing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。