不用图片也能提升语言模型的视觉理解能力，更高效。
Improving the Efficiency of Visually Augmented Language Models
- 用CLIP生成的文本表示替代图像，实现视觉增强
- 新模型BLIND-VALM在多项任务上表现与原模型相当
- 效率更高，适合资源受限场景下的视觉语言模型应用
尽管自回归语言模型（LM）表现优异，但因报告偏差，其缺乏对视觉世界及其属性的认知。现有方案通常依赖显式图像，需耗时的检索或图像生成系统。本文表明，显式图像并非必要。我们改用来自知名多模态系统CLIP的视觉语义文本表示。为公平比较，我们将使用图像检索与表示的视觉增强语言模型VALM，改造为直接使用视觉语义文本表示，形成新模型BLIND-VALM。实验显示，尽管效率显著提升且结构更简单，BLIND-VALM在视觉语言理解（VLU）、自然语言理解（NLU）和语言建模任务上性能与VALM相当。进一步在相同算力预算下扩大模型规模或预训练数据量，其在所有任务上均超越VALM。
原文摘要 · Abstract (English)
Despite the impressive performance of autoregressive Language Models (LM) it has been shown that due to reporting bias, LMs lack visual knowledge, i.e. they do not know much about the visual world and its properties. To augment LMs with visual knowledge, existing solutions often rely on explicit images, requiring time-consuming retrieval or image generation systems. This paper shows that explicit images are not necessary to visually augment an LM. Instead, we use visually-grounded text representations obtained from the well-known CLIP multimodal system. For a fair comparison, we modify VALM, a visually-augmented LM which uses image retrieval and representation, to work directly with visually-grounded text representations. We name this new model BLIND-VALM. We show that BLIND-VALM performs on par with VALM for Visual Language Understanding (VLU), Natural Language Understanding (NLU) and Language Modeling tasks, despite being significantly more efficient and simpler. We also show that scaling up our model within the compute budget of VALM, either increasing the model or pre-training corpus size, we outperform VALM for all the evaluation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。