用少量数据高效融合新模态到大模型，解决低资源模态接入难题。
Sample-efficient Integration of New Modalities into Large Language Models
- 通过超网络动态生成适配器，仅需少量样本即可适配新模态。
- 在卫星图、分子结构等新模态上，只需1/64的数据量就达同等精度。
- 适合快速拓展大模型的模态能力，尤其适用于数据稀缺的新领域。
多模态基础模型可处理多种模态，但模态空间庞大且持续演进，从头训练覆盖所有模态不可行。现有方法将新模态集成到预训练基础模型需大量成对数据，而低资源模态往往缺乏此类数据。本文提出一种样本高效的模态集成（SEMI）方法：设计一个超网络，可动态适配共享投影器——位于模态特定编码器与大语言模型（LLM）之间。该超网络在高资源模态（文本、语音、音频、视频）上训练，推理时仅需少量目标模态样本即可生成合适适配器。为提升训练多样性，引入等距变换人工扩充编码器数量。实验表明，SEMI在卫星图像、天文图像、惯性测量和分子结构等新模态的少样本集成中显著提升效率：例如，达到32样本的精度，传统方法需64倍更多数据。这使基础模型具备更广的模态覆盖潜力。
原文摘要 · Abstract (English)
Multimodal foundation models can process several modalities. However, since the space of possible modalities is large and evolving over time, training a model from scratch to encompass all modalities is unfeasible. Moreover, integrating a modality into a pre-existing foundation model currently requires a significant amount of paired data, which is often not available for low-resource modalities. In this paper, we introduce a method for sample-efficient modality integration (SEMI) into Large Language Models (LLMs). To this end, we devise a hypernetwork that can adapt a shared projector -- placed between modality-specific encoders and an LLM -- to any modality. The hypernetwork, trained on high-resource modalities (i.e., text, speech, audio, video), is conditioned on a few samples from any arbitrary modality at inference time to generate a suitable adapter. To increase the diversity of training modalities, we artificially multiply the number of encoders through isometric transformations. We find that SEMI achieves a significant boost in sample efficiency during few-shot integration of new modalities (i.e., satellite images, astronomical images, inertial measurements, and molecules) with encoders of arbitrary embedding dimensionality. For instance, to reach the same accuracy as 32-shot SEMI, training the projector from scratch needs 64$\times$ more data. As a result, SEMI holds promise to extend the modality coverage of foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。