将SAM模型改进用于自动驾驶语义分割,提升精度与效率。
AD-SAM: Fine-Tuning the Segment Anything Vision Foundation Model for Autonomous Driving Perception
- 双编码器融合全局语义与局部细节,可精准捕捉道路复杂结构。
- 在Cityscapes和BDD100K上分别达到68.1和59.5 mIoU,领先基线超20点。
- 仅需1000张标注图即保持0.607 mIoU,适合数据稀缺场景
本文提出自动驾驶语义分割专用的AD-SAM模型,基于原始Segment Anything Model(SAM)进行微调。通过引入双编码器架构,融合ViT-H的全局语义与ResNet-50的局部空间细节,并采用可变形融合模块对齐多尺度特征。解码器使用可变形注意力实现分阶段精细化预测。训练采用结合Focal、Dice、Lovasz-Softmax和Surface损失的混合损失函数,增强类别平衡性、边界精度与优化稳定性。在Cityscapes与BDD100K基准测试中,AD-SAM分别取得68.1和59.5 mIoU,较SAM、G-SAM及DeepLabV3最高提升达+22.9和+19.2 mIoU。具备强跨域泛化能力(保留率0.87,对比SAM的0.76),且学习速度提升一倍,30-40轮内收敛。仅用1000样本即可维持0.607 mIoU,表明其高度数据效率,适用于降低标注成本的自动驾驶感知任务。
原文摘要 · Abstract (English)
This paper presents the Autonomous Driving Segment Anything Model (AD-SAM), a fine-tuned vision foundation model for semantic segmentation in autonomous driving (AD). AD-SAM extends the Segment Anything Model (SAM) with a dual-encoder and deformable decoder tailored to spatial and geometric complexity of road scenes. The dual-encoder produces multi-scale fused representations by combining global semantic context from SAM's pretrained Vision Transformer (ViT-H) with local spatial detail from a trainable convolutional deep learning backbone (i.e., ResNet-50). A deformable fusion module aligns heterogeneous features across scales and object geometries. The decoder performs progressive multi-stage refinement using deformable attention. Training is guided by a hybrid loss that integrates Focal, Dice, Lovasz-Softmax, and Surface losses, improving semantic class balance, boundary precision, and optimization stability. Experiments on the Cityscapes and Berkeley DeepDrive 100K (BDD100K) benchmarks show that AD-SAM surpasses SAM, Generalized SAM (G-SAM), and a deep learning baseline (DeepLabV3) in segmentation accuracy. It achieves 68.1 mean Intersection over Union (mIoU) on Cityscapes and 59.5 mIoU on BDD100K, outperforming SAM, G-SAM, and DeepLabV3 by margins of up to +22.9 and +19.2 mIoU in structured and diverse road scenes, respectively. AD-SAM demonstrates strong cross-domain generalization with a 0.87 retention score (vs. 0.76 for SAM), and faster, more stable learning dynamics, converging within 30-40 epochs, enjoying double the learning speed of benchmark models. It maintains 0.607 mIoU with only 1000 samples, suggesting data efficiency critical for reducing annotation costs. These results confirm that targeted architectural and optimization enhancements to foundation models enable reliable and scalable AD perception.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。