用少量数据让大模型精准分割,还能自适应不同场景。
Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models
- 引入可学习的实例感知令牌,仅微调1%参数提升泛化能力。
- 无监督适配多场景,夜间等复杂环境分割精度显著提升。
- 适合大模型在真实多样场景下快速部署,无需标注数据。
视觉基础模型(VFMs)在计算机视觉任务中表现卓越,但其在语义分割中的应用面临两大挑战:一是数据规模差异,分割数据集通常远小于预训练所用数据;二是域分布偏移,现实场景多样且常未被充分覆盖。为此,我们提出Rein++,一种高效的基于VFM的分割框架,具备强泛化与有效自适应能力。Rein++包含两个组件:Rein-G实现域泛化,通过一组可学习的实例感知令牌优化特征,仅微调骨干网络1%以下参数,实现稳健泛化;Rein-A在实例和逻辑层进行无监督域适应,缓解域偏移,并引入语义迁移模块,利用Segment Anything Model的类无关能力增强目标域边界细节。整体流程先在源域(如白天场景)学习通用模型,再无标签地适配至多样目标域(如夜间场景)。大量实验表明,Rein++以高效训练显著超越现有方法,证明其作为大规模模型在复杂场景下高效、通用且可适应的分割方案的潜力。代码已开源:https://github.com/wloves/Rein。
原文摘要 · Abstract (English)
Vision Foundation Models(VFMs) have achieved remarkable success in various computer vision tasks. However, their application to semantic segmentation is hindered by two significant challenges: (1) the disparity in data scale, as segmentation datasets are typically much smaller than those used for VFM pre-training, and (2) domain distribution shifts, where real-world segmentation scenarios are diverse and often underrepresented during pre-training. To overcome these limitations, we present Rein++, an efficient VFM-based segmentation framework that demonstrates superior generalization from limited data and enables effective adaptation to diverse unlabeled scenarios. Specifically, Rein++ comprises a domain generalization solution Rein-G and a domain adaptation solution Rein-A. Rein-G introduces a set of trainable, instance-aware tokens that effectively refine the VFM's features for the segmentation task. This parameter-efficient approach fine-tunes less than 1% of the backbone's parameters, enabling robust generalization. Building on the Rein-G, Rein-A performs unsupervised domain adaptation at both the instance and logit levels to mitigate domain shifts. In addition, it incorporates a semantic transfer module that leverages the class-agnostic capabilities of the segment anything model to enhance boundary details in the target domain. The integrated Rein++ pipeline first learns a generalizable model on a source domain (e.g., daytime scenes) and subsequently adapts it to diverse target domains (e.g., nighttime scenes) without any target labels. Comprehensive experiments demonstrate that Rein++ significantly outperforms state-of-the-art methods with efficient training, underscoring its roles an efficient, generalizable, and adaptive segmentation solution for VFMs, even for large models with billions of parameters. The code is available at https://github.com/wloves/Rein.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。