用GPU加速仿真,让水下多智能体追踪更高效。
Scaling Multi Agent Reinforcement Learning for Underwater Acoustic Tracking via Autonomous Vehicles
- 开发GPU加速环境,比Gazebo快3万倍。
- 新架构在多目标、多车辆场景下误差低于5米。
- 适合大规模水下探测与智能体协同研究者。
自主车辆(AV)为水下追踪等科学任务提供了成本效益高的解决方案。强化学习(RL)在控制AV方面表现强劲,但扩展至舰队规模(对多目标或高速目标追踪至关重要)时面临挑战。多智能体强化学习(MARL)通常样本效率低下,尽管高保真仿真器Gazebo的LRAUV可实现高达100倍于实时的单机器人模拟,但在多车场景中提速有限,导致MARL训练不切实际。然而,高保真仿真对测试复杂策略和缩小仿真到现实的差距至关重要。为此,我们开发了基于GPU的加速环境,相比Gazebo实现了最高达30,000倍的加速,同时保留其动力学特性。这使得端到端的GPU训练成为可能,并可无缝迁移至Gazebo进行评估。我们还引入一种基于Transformer的架构(TransfMAPPO),学习对舰队规模和目标数量不变的策略,支持课程学习,逐步训练更大规模舰队应对更复杂场景。经过大规模GPU训练后,我们在Gazebo中进行了广泛评估,结果表明即使面对多个快速移动的目标,该方法仍能保持追踪误差低于5米。
原文摘要 · Abstract (English)
Autonomous vehicles (AVs) offer a cost-effective solution for scientific missions such as underwater tracking. Reinforcement learning (RL) has emerged as a powerful method for controlling AVs, but scaling to fleets (essential for multi-target tracking or rapidly moving targets) is challenging. Multi-Agent RL (MARL) is notoriously sample-inefficient, and while high-fidelity simulators like Gazebo's LRAUV provide up to 100x faster-than-real-time single-robot simulations, they offer little speedup in multi-vehicle scenarios, making MARL training impractical. Yet, high-fidelity simulation is crucial to test complex policies and close the sim-to-real gap. To address these limitations, we develop a GPU-accelerated environment that achieves up to 30,000x speedup over Gazebo while preserving its dynamics. This enables fast, end-to-end GPU training and seamless transfer to Gazebo for evaluation. We also introduce a Transformer-based architecture (TransfMAPPO) that learns policies invariant to fleet size and number of targets, enabling curriculum learning to train larger fleets on increasingly complex scenarios. After large-scale GPU training, we perform extensive evaluations in Gazebo, showing our method maintains tracking errors below 5m even with multiple fast-moving targets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。