用智能选题提升自动驾驶训练效率,提速近八成
Scaling Curriculum Learning For Autonomous Driving

- 将场景筛选转为无监督环境设计,按学习价值动态选题
- 99%成功率提前十亿步达成,节省77%训练时间
- 适合大规模强化学习训练,尤其在算力有限时优势显著
批量模拟器使自动驾驶强化学习可在数日内训练数千个交通场景、数十亿次交互。然而,标准的域随机化均匀采样场景,导致大量无效交互。本文提出CL4AD,首次将课程学习融入批量自动驾驶模拟器,将场景选择建模为无监督环境设计问题。引入基于成功率和行为真实性的效用函数,结合已有后悔估计函数构建课程。在GPUDRIVE上的大规模实验表明,课程学习比域随机化提前十亿步达到99%成功率,节省77%实际训练时间;优于静态与动态启发式课程(仅在最大规模例外)。算力受限下的消融实验显示样本效率提升67%。还分析了效用函数在规模下的表现及场景优先级演化过程。代码已开源。
原文摘要 · Abstract (English)
Batched simulators for autonomous driving have recently enabled training reinforcement learning (RL) agents at scale, encompassing thousands of traffic scenarios and billions of interactions within a matter of days. Although such high-throughput feeds RL algorithms faster than ever, their sample-efficiency has not kept pace: As the standard training scheme, domain randomization uniformly samples scenarios, thereby consuming a vast number of interactions on cases that contribute little to learning. Curriculum learning offers a remedy by adaptively prioritizing scenarios that matter most to policy improvement. We present CL4AD, the first integration of curriculum learning into batched autonomous driving simulators by framing scenario selection as an unsupervised environment design problem. We introduce utility functions that shape curricula based on success rates and the realism of the agent's behavior, in addition to existing regret-estimation functions. Large-scale experiments in GPUDRIVE demonstrate that curriculum learning achieves a 99% success rate a billion steps earlier than domain randomization, reducing wall-clock time by 77%, and outperforms heuristic curricula with static and dynamic attributes, with only one exception at the largest scale. An ablation under limited compute shows that curriculum learning improves sample efficiency by 67%. We also investigate how utility functions behave at scale, and how prioritized scenarios evolve during training. We release an implementation of CLForAD in GPUDRIVE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。