用密集奖励训练模型,再蒸馏到稀疏奖励策略,提升自动驾驶泛化能力。
Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policies
- 先用密集奖励训练教师模型,再将其隐状态动态蒸馏给仅用稀疏奖励训练的学生模型。
- 在未见路线测试中,学生模型成功率比教师高23%,且更安全。
- 适合追求真实部署性能的自动驾驶研究者,尤其关注泛化与安全性时。
我们研究如何在视觉自主驾驶中利用仿真器定义的密集奖励,而不继承其与实际部署指标的偏差。在CARLA等真实仿真器中,可将特权状态(如车道几何、违规行为、碰撞时间)转化为密集奖励,以稳定并加速基于模型的强化学习,但直接在这些信号上训练的策略往往过拟合,在路由完成和无碰撞超车等稀疏目标上泛化失败。本文提出奖励特权世界模型蒸馏框架:先用密集特权奖励训练一个类DreamerV3的教师代理,仅将其隐状态动态蒸馏到一个学生模型中,该学生仅使用稀疏任务奖励进行训练。师生共享相同的观测空间(语义鸟瞰图),特权信息仅通过教师奖励输入,学生不模仿教师动作或价值估计。学生的世界模型被正则化以匹配教师的隐状态动态,而其策略从零开始学习稀疏成功/失败信号。在CARLA车道保持与超车基准测试中,稀疏奖励学生优于密集奖励教师和从零开始的稀疏基线。在未见车道保持路径上,蒸馏方法使成功率相对密集教师提高约23%,同时保持相当或更好的安全性;在超车任务中,学生在训练路径上保持接近完美的表现,并在未见路径上成功率最高提升27倍,且车道保持能力改善。结果表明,密集奖励可用于学习更丰富的动态模型,同时确保部署策略严格优化于稀疏、与部署对齐的目标。
原文摘要 · Abstract (English)
We study how to exploit dense simulator-defined rewards in vision-based autonomous driving without inheriting their misalignment with deployment metrics. In realistic simulators such as CARLA, privileged state (e.g., lane geometry, infractions, time-to-collision) can be converted into dense rewards that stabilize and accelerate model-based reinforcement learning, but policies trained directly on these signals often overfit and fail to generalize when evaluated on sparse objectives such as route completion and collision-free overtaking. We propose reward-privileged world model distillation, a two-stage framework in which a teacher DreamerV3-style agent is first trained with a dense privileged reward, and only its latent dynamics are distilled into a student trained solely on sparse task rewards. Teacher and student share the same observation space (semantic bird's-eye-view images); privileged information enters only through the teacher's reward, and the student does not imitate the teacher's actions or value estimates. Instead, the student's world model is regularized to match the teacher's latent dynamics while its policy is learned from scratch on sparse success/failure signals. In CARLA lane-following and overtaking benchmarks, sparse-reward students outperform both dense-reward teachers and sparse-from-scratch baselines. On unseen lane-following routes, reward-privileged distillation improves success by about 23 percent relative to the dense teacher while maintaining comparable or better safety. On overtaking, students retain near-perfect performance on training routes and achieve up to a 27x improvement in success on unseen routes, with improved lane keeping. These results show that dense rewards can be leveraged to learn richer dynamics models while keeping the deployed policy optimized strictly for sparse, deployment-aligned objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。