首次系统评估轻量模型跨数据集泛化能力，发现ImageNet表现不靠谱。
Beyond ImageNet: Understanding Cross-Dataset Robustness of Lightweight Vision Models
- 在7个数据集上统一训练11个轻量模型，用xScore量化跨域鲁棒性。
- xScore仅需4个数据集即可预测移动端性能，且能揭示模型泛化瓶颈。
- 空间分辨率高+通道注意力更优，Transformer模块反而拖累效率。
轻量级视觉分类模型如MobileNet、ShuffleNet和EfficientNet广泛部署于移动与嵌入式设备，但其性能主要在ImageNet上评估。这引发关键问题：在ImageNet上表现优异的模型是否能在其他领域泛化？如何系统量化跨数据集鲁棒性？哪些架构元素在资源受限下持续促进泛化？本文首次对11个轻量级视觉模型（2.5M参数）进行系统评估，所有模型在固定100轮训练周期下在7个不同数据集上训练。我们提出跨数据集评分（xScore），统一衡量模型在多样化视觉领域中的性能一致性与鲁棒性。结果表明：(1) ImageNet精度无法可靠预测细粒度或医学数据集上的表现；(2) xScore可作为可扩展的移动端性能预测指标，仅需4个数据集即可估算；(3) 某些架构组件——如具有更高空间分辨率的各向同性卷积和通道注意力——促进更广泛的泛化，而基于Transformer的模块虽增加参数开销却带来有限增益。本研究提供了超越ImageNet的轻量模型可复现评估框架，明确了面向移动应用的架构设计原则，指导未来模型在多样场景下的稳健泛化开发。
原文摘要 · Abstract (English)
Lightweight vision classification models such as MobileNet, ShuffleNet, and EfficientNet are increasingly deployed in mobile and embedded systems, yet their performance has been predominantly benchmarked on ImageNet. This raises critical questions: Do models that excel on ImageNet also generalize across other domains? How can cross-dataset robustness be systematically quantified? And which architectural elements consistently drive generalization under tight resource constraints? Here, we present the first systematic evaluation of 11 lightweight vision models (2.5M parameters), trained under a fixed 100-epoch schedule across 7 diverse datasets. We introduce the Cross-Dataset Score (xScore), a unified metric that quantifies the consistency and robustness of model performance across diverse visual domains. Our results show that (1) ImageNet accuracy does not reliably predict performance on fine-grained or medical datasets, (2) xScore provides a scalable predictor of mobile model performance that can be estimated from just four datasets, and (3) certain architectural components--such as isotropic convolutions with higher spatial resolution and channel-wise attention--promote broader generalization, while Transformer-based blocks yield little additional benefit, despite incurring higher parameter overhead. This study provides a reproducible framework for evaluating lightweight vision models beyond ImageNet, highlights key design principles for mobile-friendly architectures, and guides the development of future models that generalize robustly across diverse application domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。