arXiv:2412.18409cs.CV2024-12被引 1

发现图像识别评估中单标签假设导致误判,模型实际多标签能力被低估。

The Impact of the Single-Label Assumption in Image Recognition Benchmarking

  • 引入可变top-k评估,按每图真实标签数调整评价标准
  • 315个模型测试显示,传统top-1准确率虚降11-14个百分点
  • 提出新指标ASMA和合成数据集PatchML,揭示模型隐藏的多标签能力

深度神经网络通常在每张图像仅有一个正确标签的假设下进行评估。然而,ImageNet等基准数据集中许多图像包含多个有效标签,导致评估协议与视觉数据真实复杂性不匹配。这种不匹配会惩罚模型对正确但未标注标签的预测,可能部分解释了广泛报道的图像识别性能下降,如在ImageNetV2(ImageNet的复现测试集)上出现的11至14个百分点的top-1准确率下降。这引发疑问:这些下降是模型泛化能力的真实缺陷,还是受限评估指标的人为结果?本文系统评估了多标签特性对报告准确率差距的影响。为评估单标签训练模型的多标签预测能力(MLPC),我们提出可变top-k评估,其中k等于图像的有效标签数。对315个ImageNet训练模型的分析表明,传统top-1准确率对有效但次要预测存在过度惩罚。我们还提出聚合子组模型准确率(ASMA),以更准确捕捉模型子组的多标签表现。结果显示,多标签预测能力差异显著,部分模型始终将多个正确标签排在前列。在此评估下,ImageNet与ImageNetV2之间的感知差距大幅缩小。为进一步隔离多标签识别性能与上下文线索的影响,我们引入了合成数据集PatchML,其由系统性组合的对象块构成。PatchML表明,即使在单标签监督下训练的模型,仍能识别多个物体。总体而言,这些发现揭示了单标签评估的局限性,并表明现代DNN的多标签能力远超标准指标所反映的程度。

原文摘要 · Abstract (English)

Deep neural networks (DNNs) are typically evaluated under the assumption that each image has a single correct label. However, many images in benchmarks like ImageNet contain multiple valid labels, creating a mismatch between evaluation protocols and the actual complexity of visual data. This mismatch can penalize DNNs for predicting correct but unannotated labels, which may partly explain reported accuracy drops, such as the widely cited 11 to 14 percent top-1 accuracy decline on ImageNetV2, a replication test set for ImageNet. This raises the question: do such drops reflect genuine generalization failures or artifacts of restrictive evaluation metrics? We rigorously assess the impact of multi-label characteristics on reported accuracy gaps. To evaluate the multi-label prediction capability (MLPC) of single-label-trained models, we introduce a variable top-$k$ evaluation, where $k$ matches the number of valid labels per image. Applied to 315 ImageNet-trained models, our analyses demonstrate that conventional top-1 accuracy disproportionately penalizes valid but secondary predictions. We also propose Aggregate Subgroup Model Accuracy (ASMA) to better capture multi-label performance across model subgroups. Our results reveal wide variability in MLPC, with some models consistently ranking multiple correct labels higher. Under this evaluation, the perceived gap between ImageNet and ImageNetV2 narrows substantially. To further isolate multi-label recognition performance from contextual cues, we introduce PatchML, a synthetic dataset containing systematically combined object patches. PatchML demonstrates that many models trained with single-label supervision nonetheless recognize multiple objects. Altogether, these findings highlight limitations in single-label evaluation and reveal that modern DNNs have stronger multi-label capabilities than standard metrics suggest.

图像识别多标签评估方法模型能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。