提升视觉音频融合能力，让模型听懂声音时也能看懂图。
Investigating and Enhancing Vision-Audio Capability in Omnimodal Large Language Models
- 用视觉文本部分教视觉音频部分，实现跨模态知识迁移。
- 在多模态任务上，音频查询的性能显著提升，接近文本处理水平。
- 适合研究多模态大模型、音频视觉融合的学者和开发者。
多模态大语言模型（OLLM）在融合视觉与文本方面已取得显著进展，但在处理视觉与音频的融合时仍表现不佳，尤其在使用音频查询时性能远低于文本查询。这主要源于训练过程中视觉与音频模态对齐不足，导致音频查询时对视觉信息关注不够。为此，本文提出自知识蒸馏（Self-KD）训练方法：以模型中的视觉-文本组件作为教师，视觉-音频组件作为学生，使模型能像处理文本一样理解音频。实验表明，该方法有效提升了视觉-音频能力，增强了音频与图像间的交互，显著改善了多模态任务表现。
原文摘要 · Abstract (English)
Omnimodal Large Language Models (OLLMs) have shown significant progress in integrating vision and text, but still struggle with integrating vision and audio, often exhibiting suboptimal performance when processing audio queries compared to text queries. This disparity is primarily due to insufficient alignment between vision and audio modalities during training, leading to inadequate attention to visual information when using audio queries. To mitigate this issue, we propose a Self-Knowledge Distillation (Self-KD) training method where the vision-text component of the OLLM serves as the teacher and the vision-audio component as the student. This enables the model to process audio in a manner analogous to its text processing. Our experimental results demonstrate that Self-KD is an effective method for enhancing the vision-audio capabilities of OLLMs by learning from the vision-text components, which subsequently improves the interaction between audio and images and results in improved performance on multimodal tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。