提出新模型提升音视频问答中多模态信息对齐能力
SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering
- 用可学习的源感知标记捕捉音视频元素
- 在两个数据集上准确率超越现有方法
- 适合需要精细多模态融合的任务场景
音视频问答（AVQA）是一项挑战性任务，需基于视频中的视听信息回答问题。主要难点在于理解包含视觉对象与声音源的复杂多模态场景，并将其与问题关联。本文提出源感知语义表示网络（SaSR-Net），一种新型AVQA模型。SaSR-Net利用针对不同声源的可学习标记，高效捕捉并对齐音频-视觉元素与问题内容。通过空间与时间注意力机制，简化音视频信息融合过程，以识别多模态场景中的答案。在Music-AVQA和AVQA-Yang数据集上的大量实验表明，SaSR-Net优于当前最优的AVQA方法。
原文摘要 · Abstract (English)
Audio-Visual Question Answering (AVQA) is a challenging task that involves answering questions based on both auditory and visual information in videos. A significant challenge is interpreting complex multi-modal scenes, which include both visual objects and sound sources, and connecting them to the given question. In this paper, we introduce the Source-aware Semantic Representation Network (SaSR-Net), a novel model designed for AVQA. SaSR-Net utilizes source-wise learnable tokens to efficiently capture and align audio-visual elements with the corresponding question. It streamlines the fusion of audio and visual information using spatial and temporal attention mechanisms to identify answers in multi-modal scenes. Extensive experiments on the Music-AVQA and AVQA-Yang datasets show that SaSR-Net outperforms state-of-the-art AVQA methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。