重标注ImageNet-1k数据集，发现12%标签错误，提升模型准确率。
Doomed to Re-Annotate, Forever: The ImageNet Story
- 通过人工与LLM协作反复修正，实现多标签、定位与语义属性标注
- 12%原标签错误，33.3%图像为多标签，3.8%无目标类物体
- 适用于追求高精度视觉识别的科研与工程团队
ImageNet-1k的Top-1准确率仍是视觉识别中最常用的指标。尽管数据集存在质量缺陷的问题屡被提及，但原始2012年的噪声标签仍被广泛使用。本文开展了一项全面的重标注工作，远超以往尝试，旨在获取准确且完整的ImageNet-1k验证集标注。结果是ReImageNet，包含多标签修正、目标定位、修订类别定义及语义属性（文本识别、表现形式、反光、人群、主导色）。重标注显示约12%的原始标签不正确，33.3%的图像为多标签，3.8%图像不含任何ImageNet-1k类物体。新标签使监督模型的Top-1准确率提升最高达1.2%，MLLMs提升5-6%。我们主张在如此规模下，标注无法一次完成，因错误与定义问题仅在标注过程中浮现，因此构建了以重复精炼和错误检查为核心的流水线。观察表明，人类与LLM结合并辅以合适工具，是当前该规模标注的质量上限。ImageNet-1k的问题传播至其衍生测试集，说明这是结构性问题而非单一基准特有。所有标注、类别定义、指南与分析代码已公开。项目页：https://vrg.fel.cvut.cz/reimagenet，标注：https://huggingface.co/datasets/vrg-prague/ReImageNet，代码：https://github.com/klarajanouskova/ImageNet
原文摘要 · Abstract (English)
Top-1 accuracy on ImageNet-1k remains the most commonly reported metric in visual recognition. Quality issues with the dataset have been repeatedly reported, yet the original 2012 noisy labels are still predominantly used. The paper presents a comprehensive effort, which goes well beyond prior correction attempts, towards obtaining accurate and complete ImageNet-1k validation set annotations. The result, ReImageNet, includes multilabel correction, object localization, revised class definitions, and semantic attributes (text-recognition, rendition, reflection, crowd, dominant). The reannotation reveals that approximately 12% of the original ImageNet-1k labels are incorrect, 33.3% of images are multilabel and 3.8% contain no object from an ImageNet-1k class. With the new labels, top-1 accuracy increases by up to 1.2% for supervised models and by 5-6% for MLLMs. We argue that annotation at ImageNet scale cannot realistically be completed in one pass, as errors and definitional issues are discovered only through annotating, and we build our pipeline around repeated refinement and error checking. We observed that human and LLM collaboration with appropriate tooling represents the current quality ceiling for annotation at this scale. ImageNet-1k issues propagate into its derivative test sets, indicating that the problem is structural rather than specific to any single benchmark. All annotations, class definitions, guidelines, and analysis code have been publicly released. Project page: https://vrg.fel.cvut.cz/reimagenet Annotations: https://huggingface.co/datasets/vrg-prague/ReImageNet Code: https://github.com/klarajanouskova/ImageNet
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。