arXiv:2412.10749cs.MMcs.CV2024-12AAAI被引 24

通过分块追踪声音源,提升音视频问答中目标定位的准确性。

Patch-level Sounding Object Tracking for Audio-Visual Question Answering

  • 分三路并行追踪:运动、声音和问题相关视觉块,实现精准定位。
  • 在多个标准数据集上达到与大模型相当的效果,无需大规模预训练。
  • 适合需要细粒度音视频理解的场景,如智能监控、交互式视频分析。

回答音视频场景相关的问题(即AVQA任务)正日益流行。关键挑战在于沿时间轴准确识别并追踪与问题相关的发声对象。本文提出一种新的分块级发声对象追踪(PSOT)方法。首先引入基于运动的关键词块追踪(M-KPT)模块,利用视觉运动信息识别具有显著移动的显著视觉块,这些块更可能与发声对象及问题相关。通过计算相邻帧间的分块运动强度图,构建并引导一个运动驱动的图网络。同时设计基于声音的关键词块追踪(S-KPT)模块,显式追踪发声块,其图网络邻接矩阵由音视频对应关系图正则化。M-KPT与S-KPT模块在每个时间片段内并行运行,实现对显著与发声对象的平衡追踪。基于追踪结果,进一步提出基于问题的关键词块追踪(Q-KPT)模块,保留与问题高度相关的块,确保模型聚焦于最有效线索。音视频问题特征在各模块处理过程中持续更新,并最终聚合用于答案预测。大量实验表明,该方法在标准数据集上表现优异,即使与近期基于大规模预训练的方法相比也具备竞争力。

原文摘要 · Abstract (English)

Answering questions related to audio-visual scenes, i.e., the AVQA task, is becoming increasingly popular. A critical challenge is accurately identifying and tracking sounding objects related to the question along the timeline. In this paper, we present a new Patch-level Sounding Object Tracking (PSOT) method. It begins with a Motion-driven Key Patch Tracking (M-KPT) module, which relies on visual motion information to identify salient visual patches with significant movements that are more likely to relate to sounding objects and questions. We measure the patch-wise motion intensity map between neighboring video frames and utilize it to construct and guide a motion-driven graph network. Meanwhile, we design a Sound-driven KPT (S-KPT) module to explicitly track sounding patches. This module also involves a graph network, with the adjacency matrix regularized by the audio-visual correspondence map. The M-KPT and S-KPT modules are performed in parallel for each temporal segment, allowing balanced tracking of salient and sounding objects. Based on the tracked patches, we further propose a Question-driven KPT (Q-KPT) module to retain patches highly relevant to the question, ensuring the model focuses on the most informative clues. The audio-visual-question features are updated during the processing of these modules, which are then aggregated for final answer prediction. Extensive experiments on standard datasets demonstrate the effectiveness of our method, achieving competitive performance even compared to recent large-scale pretraining-based approaches.

音视频问答目标追踪多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。