仅用文字描述就能让视觉语言模型学会新概念，无需图片或生成模型。
If you can describe it, they can see it: Cross-Modal Learning of Visual Concepts from Textual Descriptions
- 通过文本描述对齐视觉编码器特征与文本表示来注入新视觉知识。
- 单个文本描述即可高效引入新视觉概念，提升零样本性能。
- 适用于新概念学习与现有概念优化，适合需要快速扩展视觉能力的场景。
人类可基于自然语言描述和已有经验想象未知概念。受此启发，我们提出一种名为知识迁移（KT）的方法，使视觉语言模型（VLM）也能仅通过文本描述学习新概念。核心假设是：预训练VLM的知识可被重用于表征未知概念。给定新概念的文本描述后，KT通过模型反演获取视觉编码器的相关特征，并将其与文本表示对齐。不同于依赖视觉样例或外部生成模型的方法，KT在相同VLM内直接从文本注入视觉知识。在分类、分割、图文检索和图像描述等多个任务上的广泛评估表明：1）仅需单一文本描述即可高效引入新视觉概念；2）该方法同样可用于优化现有概念的表征；3）显著提升零样本VLM的性能。
原文摘要 · Abstract (English)
Humans can visualize new and unknown concepts from their natural language description, based on their experience and previous knowledge. Insipired by this, we present a way to extend this ability to Vision-Language Models (VLMs), teaching them novel concepts by only using a textual description. We refer to this approach as Knowledge Transfer (KT). Our hypothesis is that the knowledge of a pre-trained VLM can be re-used to represent previously unknown concepts. Provided with a textual description of the novel concept, KT works by aligning relevant features of the visual encoder, obtained through model inversion, to its text representation. Differently from approaches relying on visual examples or external generative models, KT transfers knowledge within the same VLM by injecting visual knowledge directly from the text. Through an extensive evaluation on several VLM tasks, including classification, segmentation, image-text retrieval, and captioning, we show that: 1) KT can efficiently introduce new visual concepts from a single textual description; 2) the same principle can be used to refine the representation of existing concepts; and 3) KT significantly improves the performance of zero-shot VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。