视觉语言训练提升概念分类知识的使用效率，但不改变其本质结构。
Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It
- 对比纯文本与视觉语言训练模型，发现后者在分类理解任务中表现更优。
- 两类模型在概念分类知识本身无显著差异，仅在问题表示方式上不同。
- 适合关注知识应用而非知识生成的研究者，尤其对多模态模型部署有启示。
视觉-语言（VL）训练是否以有意义的方式改变语言模型的语义表征？现有研究结果不一致或效果微弱。本文假设VL训练可能在词汇-概念知识，尤其是其分类组织方面产生显著影响。通过比较纯文本语言模型与经视觉语言训练的对应模型，我们发现，尽管任务为纯文本问答，后者在需要概念分类理解的任务上表现更优。通过一系列针对性的行为与表征分析，我们发现两类模型在分类知识本身无显著差异，但在处理包含分类关系与非分类关系概念的问题时，表征方式存在不同。这表明，额外的视觉语言训练并未根本改变分类知识，但提升了其在特定任务中的部署能力，即使任务呈现形式完全为语言。
原文摘要 · Abstract (English)
Does vision-and-language (VL) training change the linguistic representations of language models in meaningful ways? Most results in the literature have shown inconsistent or marginal differences, both behaviorally and representationally. In this work, we start from the hypothesis that the domain in which VL training could have a significant effect is lexical-conceptual knowledge, in particular its taxonomic organization. Through comparing minimal pairs of text-only LMs and their VL-trained counterparts, we first show that the VL models often outperform their text-only counterparts on a text-only question-answering task that requires taxonomic understanding of concepts mentioned in the questions. Using an array of targeted behavioral and representational analyses, we show that the LMs and VLMs do not differ significantly in terms of their taxonomic knowledge itself, but they differ in how they represent questions that contain concepts in a taxonomic relation vs. a non-taxonomic relation. This implies that the taxonomic knowledge itself does not change substantially through additional VL training, but VL training does improve the deployment of this knowledge in the context of a specific task, even when the presentation of the task is purely linguistic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。