评测10个大模型在低分辨率下的零样本分类能力，发现模型越大越抗模糊，预训练数据质量比数量更重要。
LR0.FM: Low-Res Benchmark and Improving Robustness for Zero-Shot Classification in Foundation Models
- 构建首个系统性低分辨率基准LR0.FM，覆盖66种模型与15个数据集
- 模型规模越大、预训练数据越优质，低分辨率下表现越强
- 提出新方法LR-TK0，不修改权重即可提升模型对模糊图像的鲁棒性
视觉-语言基础模型（FMs）在多种任务中展现出出色的零样本泛化能力，主要得益于大规模数据预训练。然而，其在低分辨率/像素化图像上的鲁棒性——这一现实场景中的常见挑战——仍缺乏研究。本文提出LR0.FM，一个全面的基准，评估10个基础模型（涵盖66个骨干网络）在15个数据集上的低分辨率零样本分类性能。我们引入一种新指标——加权聚合鲁棒性，以克服现有度量的局限。关键发现包括：(i) 模型规模与抗分辨率退化能力正相关；(ii) 预训练数据质量比数据量更关键；(iii) 经微调或高分辨率训练的模型反而在低分辨率下表现更差。分析表明，模型在低分辨率下仍能做出语义合理的预测，但输入缺失细粒度信息会更严重地影响浅层而非深层特征。基于此，我们提出简单策略LR-TK0，无需修改预训练权重即可增强模型鲁棒性。实验证明该方法在多个数据集上有效，并具有跨骨干网络与其他方法的良好泛化能力。代码已开源。
原文摘要 · Abstract (English)
Visual-language foundation Models (FMs) exhibit remarkable zero-shot generalization across diverse tasks, largely attributed to extensive pre-training on largescale datasets. However, their robustness on low-resolution/pixelated (LR) images, a common challenge in real-world scenarios, remains underexplored. We introduce LR0.FM, a comprehensive benchmark evaluating the impact of low resolution on the zero-shot classification performance of 10 FM(s) across 66 backbones and 15 datasets. We propose a novel metric, Weighted Aggregated Robustness, to address the limitations of existing metrics and better evaluate model performance across resolutions and datasets. Our key findings show that: (i) model size positively correlates with robustness to resolution degradation, (ii) pre-training dataset quality is more important than its size, and (iii) fine-tuned and higher resolution models are less robust against LR. Our analysis further reveals that the model makes semantically reasonable predictions at LR, and the lack of fine-grained details in input adversely impacts the model's initial layers more than the deeper layers. We use these insights and introduce a simple strategy, LR-TK0, to enhance the robustness of models without compromising their pre-trained weights. We demonstrate the effectiveness of LR-TK0 for robustness against low-resolution across several datasets and its generalization capability across backbones and other approaches. Code is available at https://github.com/shyammarjit/LR0.FM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。