arXiv:2504.15669cs.CV2025-04被引 1

统一多模型知识,用分层蒸馏提升少样本分割性能

UINO-FSS: Unifying Representation Learning and Few-shot Segmentation via Hierarchical Distillation and Mamba-HyperCorrelation

  • 用分层蒸馏融合DINOv2与SAM特征,构建单编码器架构
  • 1样本下在PASCAL-5i和COCO-20i上分别达80.6和64.5的mIoU
  • 适合研究少样本分割与多模态模型融合的开发者

少样本语义分割因其仅用少量标注样本即可泛化到新类别而受到关注。为应对数据稀缺问题,现有方法常引入多个基础模型以提升特征迁移能力与分割性能,但多采用双分支结构组合预训练编码器,限制了灵活性与效率。本文提出新框架UINO-FSS,关键发现早期DINOv2特征分布与SAM输出嵌入具有一致性,从而实现两模型知识在单编码器中融合。其核心组件包括:用于嵌入对齐的瓶颈适配器、基于密集相似性体积与语义嵌入的元视觉提示生成器,以及掩码解码器。通过分层跨模型蒸馏,将SAM知识有效迁移到分割器,并结合基于Mamba的4维相关性挖掘支持-查询对。在PASCAL-5$^i$和COCO-20$^i$上的实验表明,该方法在1样本设置下达到新最优性能,对应mIoU分别为80.6(+3.8%)和64.5(+4.1%),验证了统一方法的有效性。

原文摘要 · Abstract (English)

Few-shot semantic segmentation has attracted growing interest for its ability to generalize to novel object categories using only a few annotated samples. To address data scarcity, recent methods incorporate multiple foundation models to improve feature transferability and segmentation performance. However, they often rely on dual-branch architectures that combine pre-trained encoders to leverage complementary strengths, a design that limits flexibility and efficiency. This raises a fundamental question: can we build a unified model that integrates knowledge from different foundation architectures? Achieving this is, however, challenging due to the misalignment between class-agnostic segmentation capabilities and fine-grained discriminative representations. To this end, we present UINO-FSS, a novel framework built on the key observation that early-stage DINOv2 features exhibit distribution consistency with SAM's output embeddings. This consistency enables the integration of both models' knowledge into a single-encoder architecture via coarse-to-fine multimodal distillation. In particular, our segmenter consists of three core components: a bottleneck adapter for embedding alignment, a meta-visual prompt generator that leverages dense similarity volumes and semantic embeddings, and a mask decoder. Using hierarchical cross-model distillation, we effectively transfer SAM's knowledge into the segmenter, further enhanced by Mamba-based 4D correlation mining on support-query pairs. Extensive experiments on PASCAL-5$^i$ and COCO-20$^i$ show that UINO-FSS achieves new state-of-the-art results under the 1-shot setting, with mIoU of 80.6 (+3.8%) on PASCAL-5$^i$ and 64.5 (+4.1%) on COCO-20$^i$, demonstrating the effectiveness of our unified approach.

少样本分割多模态融合蒸馏Mamba

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。