arXiv:2608.08815cs.LG2026-08中稿 · the 2026 IEEE/RSJ …

用语言模型提升交通标志识别抗干扰能力,不加额外计算开销。

Distilling Vision-Language Models for Robust Traffic Sign Perception in Autonomous Vehicles

论文配图:Distilling Vision-Language Models for Robust Traffic Sign Perception in Autonomous Vehicles
图 1 · 摘自论文原文
  • 用冻结的CLIP模型生成标签语义原型,指导视觉特征学习。
  • 在四种骨干网络上对三种物理攻击均提升12.5~13.2个百分点鲁棒性。
  • 无需对抗样本或推理时模块,适合自动驾驶实际部署场景。

基于深度神经网络的交通标志识别(TSR)模型在干净数据上表现优异,但易受真实世界中的对抗攻击影响,包括阴影扰动、自然光照干扰和印刷贴片等。现有防御方法常在应对某一类攻击时提升鲁棒性,却降低其他攻击下的性能,或损害干净准确率。我们提出LAMDA(语言锚定的方向对齐模型),一种训练框架,通过将语言-视觉结构迁移至TSR模型,无需使用对抗样本且不增加推理开销。LAMDA利用冻结的OpenCLIP文本编码器,从VLM生成的标志描述和类别名中构建两个固定原型库,并在训练中通过两种互补的辅助损失监督视觉特征。推理时,适配器与原型库被移除,仅保留标准主干和分类器。在GTSRB和LISA数据集上,针对四种骨干网络和三种物理攻击类型评估,LAMDA是十种对比方法中唯一在所有攻击-骨干-数据组合下均一致提升鲁棒性的方法,阴影攻击下最高提升12.5个百分点,自然光攻击下提升13.2个百分点,且几乎全部情况下保持或提升干净准确率。

原文摘要 · Abstract (English)

Traffic sign recognition (TSR) models based on deep neural networks achieve strong clean-data performance but remain vulnerable to physically realizable adversarial attacks, including shadow perturbations, natural-light interference, and printed patches. Existing defenses often improve robustness against one attack type while degrading performance on others, and can reduce clean accuracy. We propose LAMDA (Language-Anchored Model for Direction Alignment), a training framework that transfers language-grounded structure into TSR models without using adversarial examples or adding inference-time overhead. LAMDA builds two fixed prototype banks from VLM-generated sign descriptions and class names using a frozen OpenCLIP text encoder, and uses them to supervise visual features through two complementary auxiliary losses during training. At inference, the adapter and prototype banks are discarded, leaving a standard backbone and classifier. Evaluated on GTSRB and LISA across four backbones and three physical attack types, LAMDA is the only method among ten evaluated that consistently improves robustness across all attack-backbone-dataset combinations, with gains of up to +12.5 pp under shadow attacks and +13.2 pp under natural-light attacks, while preserving or improving clean accuracy in nearly all cases.

交通标志识别鲁棒性多模态自动驾驶

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。