用视觉语言模型提炼风险注意力,提升端到端自动驾驶泛化能力。
Enhancing End-to-End Autonomous Driving with Risk Semantic Distillaion from VLM
- 从视觉语言模型中蒸馏风险语义,注入鸟瞰图特征
- 在Bench2Drive上显著提升感知与规划性能
- 适合追求高鲁棒性自动驾驶系统的研究者
自动驾驶系统在复杂驾驶场景中表现出色,但泛化能力仍是关键瓶颈,尤其在面对未见场景或陌生传感器配置时。现有方法利用视觉语言模型(VLM)处理少样本或零样本任务,但导致混合式系统出现规划不一致问题。另有些研究采用视觉-语言-动作(VLA)框架直接生成控制指令,但计算开销过大。为此,本文提出风险语义蒸馏(RSD),通过VLM为端到端自动驾驶主干网络提供风险注意力信息。具体地,引入RiskHead模块,将VLM中的因果风险估计蒸馏至鸟瞰图(BEV)特征,生成可解释的风险注意力图。该机制使BEV特征学习到更丰富细腻的风险关注表示,显著增强对空间边界和高风险物体的处理能力。实验基于Bench2Drive基准测试显示,得益于RSD增强的BEV表征,模型在复杂动态环境中感知与规划能力均有明显提升。
原文摘要 · Abstract (English)
The autonomous driving (AD) system has exhibited remarkable performance in complex driving scenarios. However, generalization is still a key limitation for the current system, which refers to the ability to handle unseen scenarios or unfamiliar sensor configurations.Related works have explored the use of Vision-Language Models (VLMs) to address few-shot or zero-shot tasks. While promising, these methods introduce a new challenge: the emergence of a hybrid AD system, where two distinct systems are used to plan a trajectory, leading to potential inconsistencies. Alternative research directions have explored Vision-Language-Action (VLA) frameworks that generate control actions from VLM directly. However, these end-to-end solutions demonstrate prohibitive computational demands. To overcome these challenges, we introduce Risk Semantic Distillation (RSD), a novel framework that leverages VLMs to enhance the training of End-to-End (E2E) AD backbones. By providing risk attention for key objects, RSD addresses the issue of generalization. Specifically, we introduce RiskHead, a plug-in module that distills causal risk estimates from Vision-Language Models into Bird's-Eye-View (BEV) features, yielding interpretable risk-attention maps.This approach allows BEV features to learn richer and more nuanced risk attention representations, which directly enhance the model's ability to handle spatial boundaries and risky objects.By focusing on risk attention, RSD aligns better with human-like driving behavior, which is essential to navigate in complex and dynamic environments. Our experiments on the Bench2Drive benchmark demonstrate the effectiveness of RSD in managing complex and unpredictable driving conditions. Due to the enhanced BEV representations enabled by RSD, we observed a significant improvement in both perception and planning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。