发现DINOv3在少样本分割中存在语义选择鸿沟，最后一层特征反而最强。
Revealing the Semantic Selection Gap in DINOv3 through Training-Free Few-Shot Segmentation
- 用类原型和格拉姆矩阵优化，零训练实现强少样本分割
- 跨域、多类任务下表现媲美复杂模型，最后一层特征最稳定
- 揭示传统选层方法失效，暴露基础模型的语义选择盲区
近期自监督视觉变压器（ViTs），如DINOv3，为密集视觉任务提供了丰富的特征表示。本研究通过无训练基线FSSDINO，利用类别特定原型和格拉姆矩阵优化，探究冻结DINOv3特征在少样本语义分割（FSS）中的内在能力。在二分类、多类及跨域（CDFSS）基准测试中，该最小化方法在最后骨干层上表现极具竞争力，可媲美需复杂解码器或测试时适应的专用方法。关键的是，我们开展基于奥数引导的层级分析，发现标准最后一层特征与全局最优中间表示之间存在显著性能差距。我们揭示了‘最安全 vs. 最优’困境：尽管奥数证明更高性能可达，但当前无监督和支持引导的选择指标始终低于最后一层基线。这表征出基础模型中的‘语义选择鸿沟’——传统启发式方法无法可靠识别高保真特征。本工作确立‘最后一层’为看似简单却强大的基线，并对DINOv3的潜在语义能力提供了严格诊断。代码公开于 https://github.com/hussni0997/fssdino。
原文摘要 · Abstract (English)
Recent self-supervised Vision Transformers (ViTs), such as DINOv3, provide rich feature representations for dense vision tasks. This study investigates the intrinsic few-shot semantic segmentation (FSS) capabilities of frozen DINOv3 features through a training-free baseline, FSSDINO, utilizing class-specific prototypes and Gram-matrix refinement. Our results across binary, multi-class, and cross-domain (CDFSS) benchmarks demonstrate that this minimal approach, applied to the final backbone layer, is highly competitive with specialized methods involving complex decoders or test-time adaptation. Crucially, we conduct an Oracle-guided layer analysis, identifying a significant performance gap between the standard last-layer features and globally optimal intermediate representations. We reveal a "Safest vs. Optimal" dilemma: while the Oracle proves higher performance is attainable, matching the results of compute-intensive adaptation methods, current unsupervised and support-guided selection metrics consistently yield lower performance than the last-layer baseline. This characterizes a "Semantic Selection Gap" in Foundation Models, a disconnect where traditional heuristics fail to reliably identify high-fidelity features. Our work establishes the "Last-Layer" as a deceptively strong baseline and provides a rigorous diagnostic of the latent semantic potentials in DINOv3.The code is publicly available at https://github.com/hussni0997/fssdino.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。