用30分钟人类数据+自对弈，让自动驾驶更像人开车。
Human-like autonomy emerges from self-play and a pinch of human data
- 在自对弈基础上加入少量人类数据作为正则化约束
- 仅需30分钟人类数据，训练15小时即完成
- 生成行为与真人轨迹高度协调，适合实际部署
自对弈强化学习近年来成为无需人类数据训练驾驶策略的方法，通过廉价的大规模模拟替代昂贵的人类驾驶示范。其主要局限是训练出的策略虽有效但行为模式与人类不兼容。以往方法依赖复杂的奖励工程和领域随机化，成本高且脆弱。本文提出将少量人类示范作为额外正则化目标，叠加于最小安全抵达奖励之上。如同炖菜中的香料，少量人类数据即可显著改善行为对齐：本方法仅使用30分钟人类示范（相比同类模仿学习减少2500倍），训练结果在保留安全性的前提下，能与独立人类轨迹良好协作，且可在单块消费级GPU上15小时内完成。视频与完整代码见https://spiced-self-play.com/。
原文摘要 · Abstract (English)
Self-play reinforcement learning has recently emerged as a way to train driving policies without any human data. It uses cheap, large-scale simulations to substitute expensive, large-scale human driving demonstrations. A key limitation of this approach is that policies trained through pure self-play can learn effective but alien driving conventions incompatible with people. Previous works attempt to mitigate such behavioral misalignments through extensive reward engineering and domain randomization, which are brittle and labor-intensive. Instead of completely discarding human demonstrations, our method treats them as a regularization objective on top of a minimal safe goal-reaching reward. Like the spice in a good stew, we find that a little human data goes a long way: our method uses only 30 minutes of human demonstrations, 2500x fewer than comparable imitation learning approaches. Resulting policies coordinate with held-out human trajectories and complete training in 15 hours on a single consumer-grade GPU. Videos and full source code are available at https://spiced-self-play.com/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。