首个面向分布外泛化的ViT神经架构搜索基准，揭示模型设计对泛化能力的关键影响。
Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights
- 构建3000个不同计算量的ViT架构，在8个OoD数据集上系统评估其泛化性能。
- 发现ID准确率与OoD准确率关联弱，现有NAS方法在预测OoD性能上效果差。
- 嵌入维度增加可提升泛化能力，参数量或浮点数等简单指标反优于复杂NAS方法。
尽管视觉变换器（ViT）已在多种机器学习任务中取得成功，但其在真实场景部署时面临关键挑战：应对分布外（OoD）数据偏移的泛化能力不足。当前研究对如何设计人工或自动化的ViT架构以提升OoD泛化仍缺乏深入理解。为此，我们提出OoD-ViT-NAS，首个专注于OoD泛化的ViT神经架构搜索系统性基准。该基准包含3000个不同计算预算的ViT架构，在8个常见的OoD数据集上进行评估。利用此基准，我们分析了影响OoD泛化的因素。结果表明：第一，ViT架构设计显著影响OoD泛化能力；第二，域内（ID）准确率常无法反映域外（OoD）准确率，提示仅优化域内性能存在风险；第三，我们首次系统评估9种免训练NAS方法在提升ViT OoD鲁棒性上的表现，发现这些方法虽在域内性能上表现优异，却普遍难以有效预测域外准确率；简单代理指标如参数量（Param）或浮点运算量（Flop）反而优于复杂免训练NAS方法。最后，我们研究了ViT架构属性对OoD泛化的影响，发现增大嵌入维度通常能提升性能。基准结果显示，不同ViT架构在OoD任务上准确率差异显著，某些分布外偏移下最高可提升11.85%。这凸显了研究ViT架构设计对提升分布外泛化能力的重要性。我们认为OoD-ViT-NAS将推动后续研究深入探索架构设计与分布外泛化之间的关系。
原文摘要 · Abstract (English)
While ViTs have achieved across machine learning tasks, deploying them in real-world scenarios faces a critical challenge: generalizing under OoD shifts. A crucial research gap exists in understanding how to design ViT architectures, both manually and automatically, for better OoD generalization. To this end, we introduce OoD-ViT-NAS, the first systematic benchmark for ViTs NAS focused on OoD generalization. This benchmark includes 3000 ViT architectures of varying computational budgets evaluated on 8 common OoD datasets. Using this benchmark, we analyze factors contributing to OoD generalization. Our findings reveal key insights. First, ViT architecture designs significantly affect OoD generalization. Second, ID accuracy is often a poor indicator of OoD accuracy, highlighting the risk of optimizing ViT architectures solely for ID performance. Third, we perform the first study of NAS for ViTs OoD robustness, analyzing 9 Training-free NAS methods. We find that existing Training-free NAS methods are largely ineffective in predicting OoD accuracy despite excelling at ID accuracy. Simple proxies like Param or Flop surprisingly outperform complex Training-free NAS methods in predicting OoD accuracy. Finally, we study how ViT architectural attributes impact OoD generalization and discover that increasing embedding dimensions generally enhances performance. Our benchmark shows that ViT architectures exhibit a wide range of OoD accuracy, with up to 11.85% improvement for some OoD shifts. This underscores the importance of studying ViT architecture design for OoD. We believe OoD-ViT-NAS can catalyze further research into how ViT designs influence OoD generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。