单领域训练导致模型丢失领域信息,影响外部数据检测效果。
Domain Feature Collapse: Implications for Out-of-Distribution Detection and Solutions
- 用信息瓶颈理论解释单域训练中领域特征消失现象
- 实验证明保留领域信息可将FPR@95提升至80%以上
- 适合关注模型泛化与微调策略的研究者
为何最先进的分布外(OOD)检测方法在单领域数据上训练时会失效?我们首次从信息论角度提供理论解释。证明监督学习在单领域数据上必然导致领域特征崩溃——表示中领域信息互信息I(x_d; z) = 0,即领域特有信息被完全丢弃。这是信息瓶颈优化的根本结果:模型在单一领域(如医学图像)训练时仅依赖类别特征,忽略领域特征,导致在检测外部样本(如在MNIST上仅达53% FPR@95)时灾难性失败。通过Fano不等式扩展分析,量化实际场景中的部分崩溃。为验证理论,我们构建了单领域数据集基准Domain Bench,证明通过预训练表征进行领域过滤可保持I(x_d; z) > 0,从而解决该问题。虽领域过滤概念简单,但其有效性为我们的信息论框架提供了强实证支持。本工作解释了令人困惑的实验现象,揭示了窄域监督学习的根本局限,并对迁移学习及微调与冻结模型的选择具有更广泛启示。
原文摘要 · Abstract (English)
Why do state-of-the-art OOD detection methods exhibit catastrophic failure when models are trained on single-domain datasets? We provide the first theoretical explanation for this phenomenon through the lens of information theory. We prove that supervised learning on single-domain data inevitably produces domain feature collapse -- representations where I(x_d; z) = 0, meaning domain-specific information is completely discarded. This is a fundamental consequence of information bottleneck optimization: models trained on single domains (e.g., medical images) learn to rely solely on class-specific features while discarding domain features, leading to catastrophic failure when detecting out-of-domain samples (e.g., achieving only 53% FPR@95 on MNIST). We extend our analysis using Fano's inequality to quantify partial collapse in practical scenarios. To validate our theory, we introduce Domain Bench, a benchmark of single-domain datasets, and demonstrate that preserving I(x_d; z) > 0 through domain filtering (using pretrained representations) resolves the failure mode. While domain filtering itself is conceptually straightforward, its effectiveness provides strong empirical evidence for our information-theoretic framework. Our work explains a puzzling empirical phenomenon, reveals fundamental limitations of supervised learning in narrow domains, and has broader implications for transfer learning and when to fine-tune versus freeze pretrained models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。