不加训练就能让音视频模型更平衡地理解两种模态。
Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models
- 推理时先分路处理音视频，再融合，避免单模态主导。
- 在三个模型上均提升音视频联合任务表现，尤其改善弱模态性能。
- 无需改模型，适合希望提升多模态公平性的研究者使用。
本工作旨在通过无需额外训练的方法增强音视频大语言模型（AV-LLMs）的平衡多模态理解能力。当前主流方法在解码器中联合处理音视频特征，虽利于统一理解，但易因训练信号不平衡导致模态偏倚，使模型过度依赖某一模态。为此，我们提出一种简单有效的推理阶段策略——分叉-合并解码（Fork-Merge Decoding, FMD）：先在早期解码层分别对音频和视频单独进行推理（分叉），再将得到的隐藏状态融合后进入后续层进行联合推理（合并）。该设计使各模态在早期阶段得以充分建模，同时促进融合时的均衡贡献。我们在 VideoLLaMA2、video-SALMONN 与 Qwen2.5-Omni 三个代表性模型上，基于三个基准数据集验证该方法。实验结果表明，其在音频、视频及音视频联合推理任务中均取得一致提升，证实了推理阶段干预对实现鲁棒高效多模态理解的有效性。
原文摘要 · Abstract (English)
The goal of this work is to enhance balanced multimodal understanding in audio-visual large language models (AV-LLMs) by addressing modality bias without additional training. In current AV-LLMs, audio and video features are typically processed jointly in the decoder. While this strategy facilitates unified multimodal understanding, it may introduce modality bias, where the model tends to over-rely on one modality due to imbalanced training signals. To mitigate this, we propose Fork-Merge Decoding (FMD), a simple yet effective inference-time strategy that requires no additional training or architectural modifications. FMD first performs modality-specific reasoning by processing audio-only and video-only inputs through the early decoder layers (fork), and then merges the resulting hidden states for joint reasoning in the remaining layers (merge). This separation allows each modality to be emphasized in the early stages while encouraging balanced contributions during integration. We validate our method on three representative AV-LLMs-VideoLLaMA2, video-SALMONN, and Qwen2.5-Omni-using three benchmark datasets. Experimental results show consistent gains in audio, video, and audio-visual reasoning tasks, highlighting the effectiveness of inference-time interventions for robust and efficient multimodal understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。