纯文本训练的LLM能隐式理解图像音频，无需专门训练。
Large Language Models Implicitly Learn to See and Hear Just By Reading
- 用文本数据训练的LLM自动获得跨模态理解能力。
- 在FSD-50K、GTZAN、CIFAR-10等数据集上实现分类准确率提升。
- 适合想复用预训练模型做多模态任务的研究者和开发者。
本文发现：仅通过文本标记训练自回归语言模型，其内部会自发形成对图像和音频的理解能力，从而实现‘读文即见、听音’。主流方法是将文本大模型微调以生成基于图像或音频嵌入的文本输出；而本研究直接输入图像块、音频波形或标记，输出分类嵌入或类别标签。实验表明，该方法在FSD-50K和GTZAN数据集上有效支持音频分类，在CIFAR-10和Fashion-MNIST及图像块上也成功实现图像分类。这表明文本大模型能学习到可被激活的通用内部结构，可用于多种任务，无需从头训练。
原文摘要 · Abstract (English)
This paper presents a fascinating find: By training an auto-regressive LLM model on text tokens, the text model inherently develops internally an ability to understand images and audio, thereby developing the ability to see and hear just by reading. Popular audio and visual LLM models fine-tune text LLM models to give text output conditioned on images and audio embeddings. On the other hand, our architecture takes in patches of images, audio waveforms or tokens as input. It gives us the embeddings or category labels typical of a classification pipeline. We show the generality of text weights in aiding audio classification for datasets FSD-50K and GTZAN. Further, we show this working for image classification on CIFAR-10 and Fashion-MNIST, as well on image patches. This pushes the notion of text-LLMs learning powerful internal circuits that can be utilized by activating necessary connections for various applications rather than training models from scratch every single time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。