用伪专家轨迹稳定离线强化学习，提升纯视觉自动驾驶安全性与成功率。
Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments
- 构建伪专家轨迹作为行为正则化信号，抑制不良驾驶行为。
- 在nuScenes仿真环境中，碰撞率显著降低，路线完成率大幅提高。
- 无需额外探索，仅用固定数据集训练，适合快速迭代验证。
端到端自动驾驶模型仅以摄像头图像为输入，直接预测未来轨迹，具备计算高效和统一优化的潜力；但其持续存在因依赖模仿学习（IL）引发的失效模式。虽然在线强化学习（RL）可缓解此问题，但基于神经渲染的仿真与大型端到端网络带来的高计算开销，使得奖励与超参数的迭代调优成本高昂。本文提出一种仅使用摄像头输入的端到端离线强化学习框架，不进行额外探索，仅在固定模拟器数据集上训练。离线强化学习虽具强数据效率和快速实验迭代优势，却易受分布外（OOD）动作过估计导致的不稳定性影响。为此，我们从专家驾驶日志中构建伪真实轨迹，作为行为正则化信号，抑制对危险或次优行为的模仿，同时稳定价值学习。训练与闭环评估在基于公开nuScenes数据集学习的神经渲染环境中进行。实验表明，该方法在碰撞率与路线完成率上显著优于模仿学习基线。代码已开源：https://github.com/ToyotaInfoTech/PEBC。
原文摘要 · Abstract (English)
End-to-end (E2E) autonomous driving models that take only camera images as input and directly predict a future trajectory are appealing for their computational efficiency and potential for improved generalization via unified optimization; however, persistent failure modes remain due to reliance on imitation learning (IL). While online reinforcement learning (RL) could mitigate IL-induced issues, the computational burden of neural rendering-based simulation and large E2E networks renders iterative reward and hyperparameter tuning costly. We introduce a camera-only E2E offline RL framework that performs no additional exploration and trains solely on a fixed simulator dataset. Offline RL offers strong data efficiency and rapid experimental iteration, yet is susceptible to instability from overestimation on out-of-distribution (OOD) actions. To address this, we construct pseudo ground-truth trajectories from expert driving logs and use them as a behavior regularization signal, suppressing imitation of unsafe or suboptimal behavior while stabilizing value learning. Training and closed-loop evaluation are conducted in a neural rendering environment learned from the public nuScenes dataset. Empirically, the proposed method achieves substantial improvements in collision rate and route completion compared with IL baselines. Our code is available at https://github.com/ToyotaInfoTech/PEBC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。