探究大规模预训练对领域泛化能力的影响,发现对齐能力比数据相似性更重要。
Is Large-Scale Pretraining the Secret to Good Domain Generalization?
- 提出对齐假说:图像与标签文本嵌入的对齐度决定领域泛化性能。
- 实验表明现有方法在预训练外数据上表现差,仅在预训练内数据上表现好。
- 强调未来需发展超越预训练对齐限制的泛化方法,适合研究领域泛化的学者。
多源领域泛化(DG)旨在利用多个源域训练模型,在未见目标域上实现高分类性能。近期方法结合网络规模预训练主干与源数据学习的新特征,显著提升了基准测试表现。然而,尚不清楚是微调方法本身进步,还是更强预训练导致的性能提升。已有研究显示感知相似性与零样本性能相关,但在DG设置中该效应有限。我们提出对齐假说:最终的DG性能高低取决于图像与类别文本嵌入的对齐程度。实验验证该假说成立,并以DomainBed数据集为基准,将评估数据分为预训练内(IP)和预训练外(OOP)。结果显示所有评估方法在DomainBed-OOP上表现不佳,而近期方法在DomainBed-IP上表现优异。综合来看,当前方法仍受限于预训练对齐,亟需发展能超越此限制的泛化技术。
原文摘要 · Abstract (English)
Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods combine robust features from web-scale pretrained backbones with new features learned from source data, and this has dramatically improved benchmark results. However, it remains unclear if DG finetuning methods are becoming better over time, or if improved benchmark performance is simply an artifact of stronger pre-training. Prior studies have shown that perceptual similarity to pre-training data correlates with zero-shot performance, but we find the effect limited in the DG setting. Instead, we posit that having perceptually similar data in pretraining is not enough; and that it is how well these data were learned that determines performance. This leads us to introduce the Alignment Hypothesis, which states that the final DG performance will be high if and only if alignment of image and class label text embeddings is high. Our experiments confirm the Alignment Hypothesis is true, and we use it as an analysis tool of existing DG methods evaluated on DomainBed datasets by splitting evaluation data into In-pretraining (IP) and Out-of-pretraining (OOP). We show that all evaluated DG methods struggle on DomainBed-OOP, while recent methods excel on DomainBed-IP. Put together, our findings highlight the need for DG methods which can generalize beyond pretraining alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。