用简单环境训练老师模型,安全高效地教学生自动驾驶决策。
Knowledge Transfer from Simple to Complex: A Safe and Efficient Reinforcement Learning Framework for Autonomous Driving Decision-Making
- 老师在轻量仿真中快速训练,复杂场景中干预危险行为
- 新算法提升样本效率,动态裁剪缓解数据不平衡问题
- 渐进式独立学习策略,适合追求安全高效的自动驾驶研究
安全高效的决策系统对自动驾驶至关重要。然而,驾驶环境的复杂性限制了规则方法和机器学习方法的效果。强化学习(RL)凭借强大的自学习能力和环境适应性,为解决这些挑战提供了可能。但训练过程中的安全与效率问题制约了其广泛应用。为此,我们提出一种新型强化学习框架——从简单到复杂协同决策(S2CD)。首先,在轻量级仿真环境中快速训练教师模型;在更复杂真实的环境中,当学生代理表现出次优行为时,教师通过评估动作价值进行干预以规避风险。我们还引入一种名为自适应裁剪近端策略优化增强(Adaptive Clipping Proximal Policy Optimization Plus)的RL算法,结合师生策略的样本,并基于样本重要性采用动态裁剪策略,提升样本效率并有效缓解数据不平衡。此外,采用KL散度作为策略约束,通过拉格朗日方法转化为无约束问题,加速学生学习。最后,渐进式脱离策略确保学生随时间逐步独立探索,克服教师局限,最大化性能。高速公路变道场景的仿真实验表明,S2CD框架提升了学习效率,降低训练成本,显著改善安全性,优于现有先进算法。该框架即使在教师表现不佳的情况下,仍能实现有效知识迁移,学生表现更优,验证了S2CD的鲁棒性与有效性。
原文摘要 · Abstract (English)
A safe and efficient decision-making system is crucial for autonomous vehicles. However, the complexity of driving environments limits the effectiveness of many rule-based and machine learning approaches. Reinforcement Learning (RL), with its robust self-learning capabilities and environmental adaptability, offers a promising solution to these challenges. Nevertheless, safety and efficiency concerns during training hinder its widespread application. To address these concerns, we propose a novel RL framework, Simple to Complex Collaborative Decision (S2CD). First, we rapidly train the teacher model in a lightweight simulation environment. In the more complex and realistic environment, teacher intervenes when the student agent exhibits suboptimal behavior by assessing actions' value to avert dangers. We also introduce an RL algorithm called Adaptive Clipping Proximal Policy Optimization Plus, which combines samples from both teacher and student policies and employs dynamic clipping strategies based on sample importance. This approach improves sample efficiency while effectively alleviating data imbalance. Additionally, we employ the Kullback-Leibler divergence as a policy constraint, transforming it into an unconstrained problem with the Lagrangian method to accelerate the student's learning. Finally, a gradual weaning strategy ensures that the student learns to explore independently over time, overcoming the teacher's limitations and maximizing performance. Simulation experiments in highway lane-change scenarios show that the S2CD framework enhances learning efficiency, reduces training costs, and significantly improves safety compared to state-of-the-art algorithms. This framework also ensures effective knowledge transfer between teacher and student models, even with suboptimal teachers, the student achieves superior performance, demonstrating the robustness and effectiveness of S2CD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。