用三阶段框架让大模型在少样本下跨域分割更准更稳。
Selective, Regularized, and Calibrated: Harnessing Vision Foundation Models for Cross-Domain Few-Shot Semantic Segmentation

- 分层选择关键特征层,动态识别最适配的视觉基础模型层级。
- 通过先验引导正则化和像素自适应校准,提升分割一致性。
- 仅微调不到2.7%参数,适合部署于资源受限的跨域场景。
视觉基础模型(VFMs)在多种视觉任务中表现优异,但在跨域少样本语义分割(CD-FSS)中仍面临挑战:每类仅有限标注样本,易导致重训练时过拟合;目标域分布与预训练数据不一致,引发跨域偏差和层敏感性。为此,提出分层示例表示适应(HERA)框架,包含三阶段:首先通过数据依赖的示例迁移风险(ETR)自适应选择最具信息量的模型层;其次采用先验引导正则化(PGR)约束特征交互,生成结构清晰的局部信号;最后通过像素级自适应校准(PAC)融合特征与优化交互图,生成一致分割掩码。该框架在不重训源数据的前提下,仅微调不足2.7%参数,实现零样本领域适应。大量实验表明,HERA在多个CD-FSS基准上超越现有最优方法超4.1 mIoU。
原文摘要 · Abstract (English)
Vision foundation models (VFMs) have achieved strong performance across various vision tasks. However, it still remains challenging to apply VFMs for cross-domain few-shot segmentation (CD-FSS), which segments objects of novel classes under domain shifts using only a few labeled exemplars. The challenge is mainly driven by two factors: (1) limited labeled exemplars per novel class relative to the scale of VFM pre-training, making the model prone to overfitting during retraining, and (2) target-domain shifts underrepresented during pre-training, inducing cross-domain inconsistency and layer-wise sensitivity. To address these issues, we propose Hierarchical Exemplar Representation Adaptation (HERA), a three-stage select-regularize-calibrate VFM-based segmentation framework that learns effectively from limited labels and adapts to novel domains without source-data retraining. We first design Hierarchical Layer Selection (HLS) to adaptively identify the most informative VFM layer using a data-dependent Exemplar Transfer Risk (ETR) computed for each candidate layer. Then, Prior-Guided Regularization (PGR) regularizes interactions on the selected representation, yielding well-structured local signals for the subsequent stage. Furthermore, Pixelwise Adaptive Calibration (PAC) combines the selected representation with the refined interaction maps to calibrate pixel-wise predictions, producing consistent masks. Together, these stages form a hierarchical select-regularize-calibrate pipeline that guides frozen VFM features in new domains while fine-tuning less than 2.7% of parameters at test time. Extensive experiments show that HERA surpasses the state of the art by more than 4.1 mIoU across multiple CD-FSS benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。