arXiv:2503.21187cs.CV2025-03被引 3

融合DINOv2与SAM2的多尺度特征,实现高效精准图像分割。

DSU-Net:An Improved U-Net Model Based on DINOv2 and SAM2 with Multi-scale Cross-model Feature Enhancement

  • 通过DINOv2引导SAM2多尺度特征融合,提升语义理解能力。
  • 轻量适配模块冻结主干,跨模态融合使模型精度显著提升。
  • 无需重新训练,适合医疗、遥感等专业领域快速部署。

尽管大规模预训练基础模型(如Meta的Segment Anything Model (SAM)系列和DINOv2)在通用图像分割上取得显著进展,但在特定领域仍受限于两大问题:大模型参数带来的高昂训练成本,以及对特定领域特征表征能力不足。本文提出一种基于DINOv2引导的SAM2多尺度特征协同框架,核心创新包括:(1) 建立DINOv2与SAM2骨干网络间的特征协同机制,由自监督模型提取的高维语义特征指导多尺度特征融合;(2) 设计轻量级适配模块与跨模态、跨层特征融合单元,在冻结基础模型参数的前提下注入跨域知识;(3) 构建基于U-Net的U形网络结构,利用注意力机制实现多粒度特征的自适应聚合解码。该框架在伪装目标检测、显著物体检测等下游任务中超越现有最先进方法,且无需昂贵训练过程,为视觉图像分割的高效部署提供技术路径,在图像分割的多种下游任务及专业领域展现显著应用价值。

原文摘要 · Abstract (English)

Despite the significant advancements in general image segmentation achieved by large-scale pre-trained foundation models (such as Meta's Segment Any-thing Model (SAM) series and DINOv2), their performance in specialized fields remains limited by two critical issues: the excessive training costs due to large model parameters, and the insufficient ability to represent specific domain characteristics. This paper proposes a multi-scale feature collabora-tion framework guided by DINOv2 for SAM2, with core innovations in three aspects: (1) Establishing a feature collaboration mechanism between DINOv2 and SAM2 backbones, where high-dimensional semantic features extracted by the self-supervised model guide multi-scale feature fusion; (2) Designing lightweight adapter modules and cross-modal, cross-layer feature fusion units to inject cross-domain knowledge while freezing the base model parameters; (3) Constructing a U-shaped network structure based on U-net, which utilizes attention mechanisms to achieve adaptive aggregation decoding of multi-granularity features. This framework surpasses existing state-of-the-art meth-ods in downstream tasks such as camouflage target detection and salient ob-ject detection, without requiring costly training processes. It provides a tech-nical pathway for efficient deployment of visual image segmentation, demon-strating significant application value in a wide range of downstream tasks and specialized fields within image segmentation.Project page: https://github.com/CheneyXuYiMin/SAM2DINO-Seg

图像分割多模态融合轻量化U-Net

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。