跨模态概念复杂度一致，揭示视觉语言模型的内在认知规律
Relative Drawing Identification Complexity is Invariant to Modality in Vision-Language Models
- 用机器教学法对比图像与坐标两种表示的教学难度
- 图像与坐标表示下概念教学规模排序高度一致
- 证明概念简单性是超越模态的本质属性，适合模型研究者
大型语言模型已具备多模态能力，许多声称通过共享表征融合不同模态。若属实，一张汽车图与一段描述其笔画的文字应在隐空间中映射到相近区域。为在黑箱访问条件下检验此假设，本文采用机器教学理论，研究教师需提供最少多少样例，学习者才能掌握特定概念。我们使用Quick, Draw!数据集中的部分对象，分别以原始位图图像和TikZ格式的笔画坐标进行呈现。结果表明，图像表示通常需要更少片段且准确率更高；但令人意外的是，在控制人类概念先验后，两种模态下的概念教学规模排序基本一致，提示概念的简单性可能是超越模态表征的固有属性。
原文摘要 · Abstract (English)
Large language models have become multimodal, and many of them are said to integrate their modalities using common representations. If this were true, a drawing of a car as an image, for instance, should map to a similar area in the latent space as a textual description of the strokes that form the drawing. To explore this in a black-box access regime to these models, we propose the use of machine teaching, a theory that studies the minimal set of examples a teacher needs to choose so that the learner captures the concept. In this paper, we evaluate the complexity of teaching vision-language models a subset of objects in the Quick, Draw! dataset using two presentations: raw images as bitmaps and trace coordinates in TikZ format. The results indicate that image-based representations generally require fewer segments and achieve higher accuracy than coordinate-based representations. But, surprisingly, the teaching size usually ranks concepts similarly across both modalities, even when controlling for (a human proxy of) concept priors, suggesting that the simplicity of concepts may be an inherent property that transcends modality representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。