轻量级框架让视觉语言模型学会听懂并说出问题，仅用数千小时语音数据即可媲美百万小时训练效果。
Speech-Omni-Lite: Portable Speech Interfaces for Vision-Language Models
- 用两个轻量模块插件式接入预训练视觉语言模型，不改变原模型结构。
- 仅用数千小时语音数据，语音问答性能接近百万小时训练的大型模型。
- 适用于资源有限但需语音交互的部署场景，如移动端或边缘设备。
尽管大规模多模态模型在多种任务上表现优异，但其性能高度依赖海量多模态数据且计算成本高昂。本文提出 Speech-Omni-Lite，一种低成本框架，可在不破坏预训练视觉-语言（VL）模型原有性能的前提下，为其添加语音理解与生成能力。具体地，通过在冻结的VL骨干网络上插入两个轻量、可训练的插件模块——语音投影器和语音标记生成器，实现语音功能扩展。为缓解语音问答语料稀缺问题，提出一种低成本数据构建策略，从现有自动语音识别（ASR）的语音-文本对中生成包含问题-文本答案-文本-语音的QTATS数据，用于高效训练语音生成。实验表明，即使仅有数千小时的语音训练数据，Speech-Omni-Lite仍能达到优异的语音问答性能，与在数百万小时语音数据上训练的大型模型相当。此外，所学语音模块在不同VL骨干网络间具有强迁移能力。
原文摘要 · Abstract (English)
While large-scale omni-models have demonstrated impressive capabilities across various modalities, their strong performance heavily relies on massive multimodal data and incurs substantial computational costs. This work introduces Speech-Omni-Lite, a cost-efficient framework for extending pre-trained Visual-Language (VL) backbones with speech understanding and generation capabilities, while fully preserving the backbones' vision-language performance. Specifically, the VL backbone is equipped with two lightweight, trainable plug-and-play modules, a speech projector and a speech token generator, while keeping the VL backbone fully frozen. To mitigate the scarcity of spoken QA corpora, a low-cost data construction strategy is proposed to generate Question-Text Answer-Text-Speech (QTATS) data from existing ASR speech-text pairs, facilitating effective speech generation training. Experimental results show that, even with only thousands of hours of speech training data, Speech-Omni-Lite achieves excellent spoken QA performance, which is comparable to omni-models trained on millions of hours of speech data. Furthermore, the learned speech modules exhibit strong transferability across VL backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。