arXiv:2605.16864cs.CVcs.AI2026-05中稿 · CVPR

用评估指标筛选互补视觉模型特征,提升分割精度。

Metric-Guided Feature Fusion of Visual Foundation Models for Segmentation Tasks

论文配图:Metric-Guided Feature Fusion of Visual Foundation Models for Segmentation Tasks
图 1 · 摘自论文原文
  • 基于结构一致性和边缘保真度设计无标签评估指标
  • 在多个分割任务上显著优于基线,边界更准、语义更优
  • 无需复杂结构改动,单阶段训练即可应用

尽管大规模视觉基础模型(VFMs)在语义理解上表现卓越,但在实例感知的密集预测任务中仍表现不足。不同模型存在差异性表征偏差:例如可提示分割模型(如SAM2)关注细粒度区域边界,而自监督模型(如DINOv3)强调对象级结构。这表明融合互补特征具有潜力。然而,简单拼接多模型特征通常无法带来可靠提升,且如何有效利用互补特征的可解释原则尚不明确。本文提出一种基于度量的引导方法,通过显式评估分数选择并聚合不同VFMs的互补特征。具体地,我们在特征空间中设计了两类无标签度量——结构一致性与边缘保真度,用于评估各类编码器特征。依据这些评分,我们识别出边强与结构强的编码器对,并采用主-辅融合方案进行集成。该融合无需复杂架构调整,仅需单阶段训练。相比基线模型,本方法在多个密集预测任务中实现持续性能提升,兼具更好的对象级语义和更精确的边界定位。代码已开源。

原文摘要 · Abstract (English)

Although large-scale visual foundation models (VFMs) achieve remarkable performance in semantic understanding, they still underperform in instance-aware dense prediction tasks. They exhibit different biases in representation: for instance, promptable segmentation models (e.g., SAM2) focus on fine-grained region boundaries, while self-supervised models (e.g., DINOv3) emphasize object-level structure. This observation highlights the potential of combining complementary features from different VFMs to enhance downstream dense prediction tasks. However, naive multi-VFM fusion seldom leads to reliable gains, and interpretable principles for leveraging their complementary features are still underexplored. In this work, we propose a metric-guided approach that effectively selects and aggregates complementary features from different VFMs based on explicit assessment scores. Specifically, we design a suite of label-free metrics in feature space across two aspects, Structural Coherence and Edge Fidelity, to assess features of VFM encoders. Guided by these scores, we identify complementary edge-strong and structure-strong encoder pairs, and integrate them via a master-auxiliary fusion scheme. This feature fusion requires no complex architectural changes and is trained only in a single stage. Our model shows consistent performance gains across multiple dense prediction tasks compared with the baselines, with better object-level semantics and more accurately localized boundaries. The code is available at {https://github.com/gyc-code/metric-guided-fusion}.

图像分割特征融合视觉模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。