验证视觉Transformer在噪声图像下的泛化能力,发现其对遮挡有强鲁棒性。
Resilience of Vision Transformers for Domain Generalisation in the Presence of Out-of-Distribution Noisy Images
- 用网格遮挡和零样本分割生成可控的分布外测试数据
- BEIT在PACS上保持94%准确率,比CNN等高出37%
- 自注意力机制使模型依赖全局特征,提升抗干扰能力
现代AI模型在受控环境下表现优异,但在现实场景中因数据分布突变而失效,这正是领域泛化(DG)的挑战。本文系统评估了基于掩码图像建模预训练的BEIT架构在合成分布外(OOD)噪声和遮挡下的表现。通过网格遮挡(25%、50%、75%)结合Segment Anything与Grounding DINO实现精准物体定位,构建新型OOD测试基准。在PACS、Office-Home、DomainNet三个基准上,尽管存在显著遮挡,BEIT仍保持94%(PACS)和87%(Office-Home)准确率,优于CNN及其他视觉变压器达37%。分析显示,自注意力距离表明其对全局特征的依赖是抗扰性的原因。此外,当遮挡破坏物体形状时,性能急剧下降——外部网格遮挡导致68%准确率下降,内部遮挡仅22%。本工作提出两种关键贡献:(1)一种可扩展的可控噪声生成方法;(2)实证证明掩码图像建模与自注意力机制能学习不变特征,增强领域泛化能力。研究弥合了实验室训练与真实部署之间的差距,为构建可靠不确定性应对系统提供蓝图。
原文摘要 · Abstract (English)
Modern AI models excel in controlled settings but often fail in real-world scenarios where data distributions shift unpredictably - a challenge known as domain generalisation (DG). This paper tackles this limitation by rigorously evaluating vision tramsformers, specifically the BEIT architecture which is a model pre-trained with masked image modelling (MIM), against synthetic out-of-distribution (OOD) benchmarks designed to mimic real-world noise and occlusions. We introduce a novel framework to generate OOD test cases by strategically masking object regions in images using grid patterns (25\%, 50\%, 75\% occlusion) and leveraging cutting-edge zero-shot segmentation via Segment Anything and Grounding DINO to ensure precise object localisation. Experiments across three benchmarks (PACS, Office-Home, DomainNet) demonstrate BEIT's known robustness while maintaining 94\% accuracy on PACS and 87\% on Office-Home, despite significant occlusions, outperforming CNNs and other vision transformers by margins of up to 37\%. Analysis of self-attention distances reveals that the BEIT dependence on global features correlates with its resilience. Furthermore, our synthetic benchmarks expose critical failure modes: performance degrades sharply when occlusions disrupt object shapes e.g. 68\% drop for external grid masking vs. 22\% for internal masking. This work provides two key advances (1) a scalable method to generate OOD benchmarks using controllable noise, and (2) empirical evidence that MIM and self-attention mechanism in vision transformers enhance DG by learning invariant features. These insights bridge the gap between lab-trained models and real-world deployment that offer a blueprint for building AI systems that generalise reliably under uncertainty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。