提出并行三模态注意力机制，提升音视频问答准确性
Q-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answering
- 用文本引导的并行三模态注意力替代深层串行融合
- 在三个基准上达最新水平，MUSIC-AVQA-R提升显著
- 适合需要高效多模态推理的场景，如智能客服、视频理解
音视频问答（AVQA）要求对视频和同步音频进行联合推理，扩展了传统视觉问答。然而，许多系统依赖于文本、视频、音频间深层堆叠的自注意力与交叉注意力，这种顺序融合可能导致细微跨模态线索丢失，错误在层层传递中累积。本文提出Q-TriM，采用浅层并行的多模态融合方式，设计了一种新颖的注意力框架，使视频和音频以文本为条件进行注意力计算。由此获得标准交叉注意力输出外，还生成了三模态注意力表示——查询、键、值分别来自不同模态。这些表示在单个阶段并行结合，避免深层堆叠带来的信息损失与误差累积。Q-TriM在三个AVQA基准上取得当前最佳性能，尤其在MUSIC-AVQA-R上表现突出，体现其鲁棒性与分布外泛化能力。代码已开源。
原文摘要 · Abstract (English)
Audio-Visual Question Answering (AVQA) extends classical VQA by requiring joint reasoning over video and synchronized audio. However, many AVQA systems rely on deeply stacked layers of self- and cross attention across text, video, and audio. Such sequential stacking may incur loss of information such as subtle inter-modal cues over the layers, causing errors to accumulate across sequential attention layers during the fusion. We introduce Q-TriM which performs multi-modal fusion in a shallow and parallel manner instead of a deep and sequential manner. For Q-TriM, we propose a novel framework for attention operation incorporating video and audio conditioned on text. As a result, we obtain not only standard cross attention outputs but also Tri-Modal Attention representations in which Query, Key, and Value come from distinct modalities. These attention representations are combined in parallel at a single stage, thus avoiding the multi-modal fusion with deep stacks in order to mitigate error accumulation and depth-induced issues. Q-TriM achieves state-of-the-art performance on three AVQA benchmarks, including substantial gains on MUSIC-AVQA-R, which demonstrates its robustness and out-of-distribution generalization. Code is available at https://github.com/Sunghun95/Q-TriM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。