arXiv:2609.04242eess.AScs.CV2026-09

不训练模型,用语音转写让视觉模型听懂说话。

Training-Free Speech-Centric Omni Understanding with Frozen VLMs

论文配图:Training-Free Speech-Centric Omni Understanding with Frozen VLMs
图 1 · 摘自论文原文
  • 用Whisper提取带时间戳的语音文本,经语言接口输入冻结的视觉模型。
  • 在56个任务上表现接近原生多模态模型,多语言语音理解显著提升。
  • 保留原始模型图像理解与推理能力,适合想快速增强语音理解的用户。

音频-视觉理解仍具挑战性,因模型需联合解析口语内容、视觉事件及其时间关系。现有全模态模型通常引入专用音频编码器,依赖昂贵的音视频文本训练,使全模态能力绑定特定视觉语言模型(VLM)主干,可能削弱其原有视觉与推理能力。我们提出无需训练的全模态框架TFO,将冻结的VLM转换为以语音为中心的全模态模型,无需架构修改或多模态对齐。TFO利用Whisper提取置信度过滤的带时间戳文本,通过VLM现有的语言接口输入,同时保持其视觉路径不变。在56个基准和21种语言的匹配对比中,TFO在音视频理解上表现相当,提升了所有五种模型设置下的平均音频仅性能,并实现显著的多语言语音增益。冻结VLM还普遍保留更强的图像/视频理解、视觉定位、编码、数学推理及医学问答能力,优于对应原生全模态检查点。结果表明,通过模块化音频到语言路由,可高效获得强语音中心的全模态理解,无需代价高昂的主干特异性训练。

原文摘要 · Abstract (English)

Audio-visual understanding remains challenging because models must jointly interpret spoken content, visual events, and their temporal relationships. Existing omni models typically introduce dedicated audio encoders and rely on expensive audio-video-text training, tightly coupling omni capability to specific VLM backbones and potentially weakening their existing visual and reasoning abilities. This raises three questions: whether native omni training is necessary for every new VLM, whether speech-centric omni capability can be added while preserving the original backbone, and where richer acoustic representations remain essential. We introduce Training-Free Omni (TFO), a plug-and-play framework that converts a frozen VLM into a speech-centric omni model without architectural modification, or multimodal re-alignment. TFO uses Whisper to extract confidence-filtered, timestamped transcripts and routes them through the VLM's existing language interface, while leaving its visual pathway unchanged. Across matched comparisons with native omni models on 56 benchmarks and 21 languages, TFO is competitive on audio-visual understanding, improves average audio-only performance across all five model settings, and achieves substantial multilingual speech gains. Freezing the VLM also generally preserves stronger image/video understanding, visual grounding, coding, mathematical reasoning, and medical question answering than the corresponding native omni checkpoints. These results show that strong speech-centric omni understanding can often be obtained through modular audio-to-language routing rather than costly backbone-specific training.

语音理解多模态零样本Whisper

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。