测试视觉概念迁移能力,揭示模型在复杂变换下的理解短板。
VisAnalog: A Diagnostic Suite for Visual Concept Transfer on Natural Images

- 设计可控的图像变换任务,评估模型对概念属性的保持与迁移能力。
- 模型准确率随变换步骤增加显著下降,远低于人类表现。
- 发现关系推理是主要瓶颈,适合研究视觉推理与通用模型性能的学者。
衡量视觉概念学习的有效性不仅在于单图识别,更在于模型能否在变换中保持并操控概念级属性,并将其迁移到新场景。本文提出 VisAnalog,一个针对自然图像的概念迁移诊断套件。每个样本为 $A o B::C o ?$ 形式:图像 $B$ 与隐藏目标图像 $D$ 分别通过相同确定性变换序列作用于源图像 $A$ 与 $C$。给定 $A$、$B$、$C$,模型需从多项选择中推断 $D$。该基准包含 617 个经人工验证的问题,涵盖 1 至 4 步变换,如缩放、象限交换、旋转、翻转和色相旋转。在主流专有与开源视觉语言模型上,端到端准确率显著低于直接显示 $D$ 时的极值准确率,且随变换深度增加急剧下降;而人类表现接近天花板。程序条件评估进一步分离了关系推理与变换应用错误,表明从 $A o B$ 推断视觉关系是主要瓶颈,多步复杂情况中才显现应用错误。数据集已公开于 https://huggingface.co/datasets/zli99/VisAnalog。
原文摘要 · Abstract (English)
A useful test of visual concept learning is not just whether a model can recognize a concept in a single image, but whether it can preserve and manipulate concept-level properties under transformation and transfer them to new scenes. We introduce VisAnalog, a controlled suite for this setting on natural images. Each example instantiates $A\!:\!B::C\!:\,?$: images $B$ and a hidden target image $D$ are produced by applying the same deterministic transformation sequence to source images $A$ and $C$. Given $A$, $B$, and $C$, a model must answer a multiple-choice question about $D$. The benchmark contains 617 human-validated questions spanning one- to four-step transformations such as zoom, quadrant swap, rotation, flip, and hue rotation. Across strong proprietary and open-source VLMs, end-to-end accuracy is substantially lower than oracle accuracy when $D$ is directly shown, and degrades sharply as transformation depth increases, while human performance remains near the ceiling. A program-conditioned evaluation further separates failures of relation inference from failures of transformation application, showing that inferring the visual relation from $A \rightarrow B$ is the dominant bottleneck, with additional application errors emerging on harder multi-step cases. The dataset is publicly available at https://huggingface.co/datasets/zli99/VisAnalog.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。