提出新方法缓解模型过度追求不变性导致的泛化能力下降问题。
DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalization
- 通过对比学习与随机掩码补偿不变性约束，防止特征信息丢失。
- 在12个数据集上验证，有效提升多种模型在分布外场景下的性能。
- 适合关注模型鲁棒性与泛化能力的研究者参考。
分布外泛化是模型在远离训练数据分布的场景下仍需保持良好性能的关键挑战。当前主流方法为不变性学习（IL），通过强约束使模型聚焦于不变特征而非虚假关联。然而，过强的约束可能导致特征空间过正则化，丢失重要细节，即“过度不变性”问题。本文首次理论定义该现象，并发现其存在于多种经典IL方法中。为此，提出多样性不变学习（DivIL），引入无监督对比学习和随机掩码机制，对冲不变性约束的负面影响，可适配多种现有IL框架。在跨模态的12个数据集、6种经典模型上实验验证，证明该方法能有效缓解过度不变性并提升泛化性能。
原文摘要 · Abstract (English)
Out-of-distribution generalization is a common problem that expects the model to perform well in the different distributions even far from the train data. A popular approach to addressing this issue is invariant learning (IL), in which the model is compiled to focus on invariant features instead of spurious features by adding strong constraints during training. However, there are some potential pitfalls of strong invariant constraints. Due to the limited number of diverse environments and over-regularization in the feature space, it may lead to a loss of important details in the invariant features while alleviating the spurious correlations, namely the over-invariance, which can also degrade the generalization performance. We theoretically define the over-invariance and observe that this issue occurs in various classic IL methods. To alleviate this issue, we propose a simple approach Diverse Invariant Learning (DivIL) by adding the unsupervised contrastive learning and the random masking mechanism compensatory for the invariant constraints, which can be applied to various IL methods. Furthermore, we conduct experiments across multiple modalities across 12 datasets and 6 classic models, verifying our over-invariance insight and the effectiveness of our DivIL framework. Our code is available at https://github.com/kokolerk/DivIL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。