测试视觉模型是否像人一样逐步理解难度,发现多数模型表现相似。
Do Vision Models Develop Human-Like Progressive Difficulty Understanding?
- 构建含100类、10属性、3难度等级的生成图像数据集
- 80%-90%情况下模型答题模式符合人类渐进式认知规律
- 提出自适应测试法,用更少步骤评估模型真实水平
人类在答题时遵循一定规律:若答错简单题(如2×3),大概率也答错难题(如2×3×4);若答对难题,则简单题通常也能答对。否则可能为记忆。当前视觉识别模型是否具备类似学习能力?本文研究图像分类任务中模型响应模式。由于真实图像无难度标签,我们利用生成模型构建一个包含100类别、10属性、3难度等级的数据集:每个类别(如狗)和属性(如遮挡)生成从易到难的图像(如无遮挡、部分可见)。结果表明,大多数模型在80%-90%时间内表现出与人类相似的渐进式响应模式。基于此特性,我们设计一种新型自适应评估方法——类似GRE的动态测试:根据模型当前表现决定下一阶段测试图像,使其跳过过于简单或困难的题目,从而以更少步骤准确评估整体性能。
原文摘要 · Abstract (English)
When a human undertakes a test, their responses likely follow a pattern: if they answered an easy question $(2 \times 3)$ incorrectly, they would likely answer a more difficult one $(2 \times 3 \times 4)$ incorrectly; and if they answered a difficult question correctly, they would likely answer the easy one correctly. Anything else hints at memorization. Do current visual recognition models exhibit a similarly structured learning capacity? In this work, we consider the task of image classification and study if those models' responses follow that pattern. Since real images aren't labeled with difficulty, we first create a dataset of 100 categories, 10 attributes, and 3 difficulty levels using recent generative models: for each category (e.g., dog) and attribute (e.g., occlusion), we generate images of increasing difficulty (e.g., a dog without occlusion, a dog only partly visible). We find that most of the models do in fact behave similarly to the aforementioned pattern around 80-90% of the time. Using this property, we then explore a new way to evaluate those models. Instead of testing the model on every possible test image, we create an adaptive test akin to GRE, in which the model's performance on the current round of images determines the test images in the next round. This allows the model to skip over questions too easy/hard for itself, and helps us get its overall performance in fewer steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。