让自动驾驶模型既懂交通语义又预判后果,实现更安全灵活的决策。
Distill to Think, Foresee to Act: Cognitive-Physical Reinforcement Learning for Autonomous Driving

- 用视觉语言模型知识蒸馏提升感知理解,零成本保留认知能力。
- 构建自回归场景模型预测未来路况,直接生成安全评估指标。
- 支持用户用自然语言控制驾驶意图,适合需要高安全性与可解释性的场景。
当前端到端自动驾驶模型受限于模仿学习的行为天花板。尽管强化学习可带来更智能的自主性,但缺乏两个关键基础设施:(1) 理解交通语义与驾驶意图的认知基础,(2) 能预判行为后果的前瞻性物理环境。为此,我们提出 CoPhy 框架。为实现‘以知促思’,我们将视觉语言模型(VLM)知识蒸馏至鸟瞰图(BEV)编码器,并完全移除 VLM,保留认知能力且推理成本为零,同时将认知通道开放为可插拔接口,支持用户语言指令。为实现‘预见而行’,我们构建了一个自回归的 BEV 世界模型,显式预测候选动作下的未来语义地图,作为可解释的物理沙盒,直接导出安全指标。在此双基础设施基础上,我们通过改进的 GRPO 算法优化驾驶策略,采用双重奖励机制:来自 BEV 滚动推演的物理奖励强制执行硬性安全约束,来自语言对齐评分器的认知奖励确保意图一致性。大量实验表明,CoPhy 在 NAVSIM v1 与 v2 基准上均达到顶尖性能,并通过认知引导的场景合规性和用户定义语言指令实现了更安全的驾驶与灵活意图控制。
原文摘要 · Abstract (English)
Current end-to-end autonomous driving models are fundamentally constrained by the behavioral cloning ceiling of imitation learning. While reinforcement learning offers a path to smarter autonomy, it demands two missing pieces of infrastructure: (1) a cognitive foundation that understands traffic semantics and driving intent, and (2) a foresighted physical environment that can anticipate the consequences of candidate actions. To this end, we propose CoPhy, a CognitivePhysical reinforcement learning framework for autonomous driving. To distill to think, we distill VLM knowledge into the BEV encoder and then discard the VLM entirely, retaining cognitive ability at zero inference cost while releasing the cognitive channel as a pluggable interface for optional human language commands. To foresee to act, we build an auto-regressive BEV world model that explicitly predicts future semantic maps conditioned on candidate actions, serving as an interpretable physical sandbox from which safety metrics are directly derived. Built upon this dual infrastructure, we optimize the driving policy via GRPO with a novel dual-reward mechanism: a physical reward derived from BEV rollouts enforces hard safety constraints, while a cognitive reward from a language-aligned scorer ensures intent compliance. Extensive experiments demonstrate that CoPhy not only achieves state-of-the-art results on NAVSIM v1 and v2 benchmarks, but also enables safer driving via cognitively informed scene compliance and flexible intent control through user-defined language instructions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。