通过分离视觉模型中的因果与非因果特征,提升跨域语义分割泛化能力。
Causal-Tune: Mining Causal Factors from Vision Foundation Models for Domain Generalized Semantic Segmentation
- 基于频域分析识别并分离特征中的因果与非因果成分。
- 在雪天条件下提升4.8% mIoU,显著改善恶劣环境下的分割性能。
- 适合需要强泛化能力的自动驾驶、遥感等跨域视觉任务。
使用少量参数微调视觉基础模型(VFMs)在跨域语义分割(DGSS)中表现出色。现有方法多通过轻量适配器或优化中间特征来增强泛化,但忽略了长期预训练模型中存在的伪影问题,这些伪影会干扰有效表示的利用,进而降低性能。受因果机制启发,我们发现这些伪影与非因果因素相关,通常位于模型特征频谱的低频和高频部分。本文首次系统分析了VFMs中特征的因果与非因果因素,提出Causal-Tune:一种新颖的微调策略,显式提取因果因素并抑制非因果因素。具体而言,通过离散余弦变换(DCT)获取各层特征的频谱,并用高斯带通滤波器将其分为因果与非因果分量;引入一组频域可学习的因果感知标记以优化因果分量,非因果分量则被丢弃;最后通过逆DCT将特征重构回空间域传递至下一层。大量跨域实验表明,该方法在极端天气下表现优异,尤其在雪天条件下相比基线提升4.8% mIoU。
原文摘要 · Abstract (English)
Fine-tuning Vision Foundation Models (VFMs) with a small number of parameters has shown remarkable performance in Domain Generalized Semantic Segmentation (DGSS). Most existing works either train lightweight adapters or refine intermediate features to achieve better generalization on unseen domains. However, they both overlook the fact that long-term pre-trained VFMs often exhibit artifacts, which hinder the utilization of valuable representations and ultimately degrade DGSS performance. Inspired by causal mechanisms, we observe that these artifacts are associated with non-causal factors, which usually reside in the low- and high-frequency components of the VFM spectrum. In this paper, we explicitly examine the causal and non-causal factors of features within VFMs for DGSS, and propose a simple yet effective method to identify and disentangle them, enabling more robust domain generalization. Specifically, we propose Causal-Tune, a novel fine-tuning strategy designed to extract causal factors and suppress non-causal ones from the features of VFMs. First, we extract the frequency spectrum of features from each layer using the Discrete Cosine Transform (DCT). A Gaussian band-pass filter is then applied to separate the spectrum into causal and non-causal components. To further refine the causal components, we introduce a set of causal-aware learnable tokens that operate in the frequency domain, while the non-causal components are discarded. Finally, refined features are transformed back into the spatial domain via inverse DCT and passed to the next layer. Extensive experiments conducted on various cross-domain tasks demonstrate the effectiveness of Causal-Tune. In particular, our method achieves superior performance under adverse weather conditions, improving +4.8% mIoU over the baseline in snow conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。