发现视觉语言模型能通过绑定ID实现图文关联，10秒理解跨模态映射机制。
Investigating Mechanisms for In-Context Vision Language Binding
- 设计合成数据集，让模型将3D物体图像与文本描述配对
- 实验显示物体图像与对应文字共享唯一绑定ID
- 揭示模型在上下文内建立图文关联的内在机制，适合研究多模态推理者
为理解提示，视觉-语言模型（VLMs）需感知图像、理解文本，并在两种模态间建立关联。例如，给定一张‘红色玩具车’的图像，模型应将其与‘车’‘红色玩具’‘红色物体’等短语关联。Feng和Steinhardt在大语言模型中提出绑定ID机制，即实体与其属性词共享绑定ID。本文通过合成数据集和任务，研究该机制在视觉-语言模型中的适用性，要求模型将图像中的3D物体与文本描述进行配对。实验表明，视觉-语言模型会为物体的图像标记和其文本引用分配唯一的绑定ID，从而实现上下文内的跨模态关联。
原文摘要 · Abstract (English)
To understand a prompt, Vision-Language models (VLMs) must perceive the image, comprehend the text, and build associations within and across both modalities. For instance, given an 'image of a red toy car', the model should associate this image to phrases like 'car', 'red toy', 'red object', etc. Feng and Steinhardt propose the Binding ID mechanism in LLMs, suggesting that the entity and its corresponding attribute tokens share a Binding ID in the model activations. We investigate this for image-text binding in VLMs using a synthetic dataset and task that requires models to associate 3D objects in an image with their descriptions in the text. Our experiments demonstrate that VLMs assign a distinct Binding ID to an object's image tokens and its textual references, enabling in-context association.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。