探究视觉语言模型如何绑定概念，发现复杂度是关键瓶颈。
How can embedding models bind concepts?
- 用分解式绑定函数分析模型对多对象场景的表示机制。
- 大模型在充足数据下能实现低复杂度乘法绑定，支持新组合泛化。
- 适合研究多模态表示、概念绑定与模型可解释性的读者。
人类能轻松判断多物体场景中颜色与形状的对应关系，这种能力称为概念绑定。视觉-语言嵌入模型如CLIP虽能识别单一概念，却难以表示概念间的归属关系。尽管CLIP在跨模态检索中表现如‘概念袋’，但其图像和文本嵌入中仍可恢复对象信息。本文通过绑定函数研究这一矛盾：发现场景嵌入可加性分解为对象表示，解释了单模态探测器为何能恢复对象信息。然而，CLIP的绑定函数复杂度高，阻碍了图像与文本编码器学习共享且泛化的绑定机制。我们进一步验证该限制是否本质存在——答案是否定的。在从零训练的受控Transformer模型中，充分的数据覆盖可使低复杂度绑定函数出现，表现为概念间的乘法交互，从而实现系统性泛化。代码已公开于https://github.com/oshapio/binding-concepts-complexity。
原文摘要 · Abstract (English)
Humans easily determine which color belongs to which shape in multi-object scenes, an ability known as concept binding. Vision-language embedding models such as CLIP struggle with binding: they recognize individual concepts but fail to represent which concepts form which objects. Although CLIP behaves like a bag-of-concepts model in cross-modal retrieval, object information is recoverable from its image and text embeddings separately. We study this tension through the binding function, which maps concepts to scene embeddings. We find that scene embeddings decompose additively into object representations, explaining why uni-modal probes can recover object information. However, CLIP's binding function is high-complexity, which likely prevents the image and text encoders from learning a shared binding mechanism that generalizes to unseen concept combinations. We then ask whether this limitation is fundamental. We show that it is not. In controlled transformer models trained from scratch, binding generalization emerges with sufficient data coverage. These models learn low-complexity binding functions characterized by multiplicative interactions between concepts, enabling systematic generalization. Code is publicly available at https://github.com/oshapio/binding-concepts-complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。