让音频更智能:动态筛选并融合音视频文本信息,提升视频定位精度。
Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval
- 基于伪标签训练音频重要性预测器,自动识别并加权有用音频
- 在局部、事件、全局三层融合音视频,充分挖掘互补信息
- 支持无音频推理,适合真实场景中音频缺失的应用
视频时刻检索(VMR)旨在定位与给定查询语义相关的特定视频片段。现有方法多仅关注视觉与文本模态,忽略重要的音频信息。尽管部分工作尝试联合处理音视频文本,但通常对各模态等同对待,缺乏细粒度交互。这不切实际:并非所有音频都有助于定位,某些视频音频可能仅为噪声或背景声。为此,我们提出重要性感知多粒度融合模型(IMG),动态选择性聚合音视频文本上下文。首先,结合文本引导分别融合视觉与音频后,设计伪标签监督的音频重要性预测器,预测音频重要性得分并加权,以抑制噪声干扰;其次,设计多粒度音频融合模块,在局部、事件和全局层面自适应融合音视频,全面捕捉互补上下文;进一步提出跨模态知识蒸馏策略,解决推理时音频缺失问题。为评估方法,构建新数据集Charades-AudioMatter,从原Charades-STA中人工筛选并重构与音频相关样本。大量实验验证其有效性,实现音视频融合类方法在VMR上的最新性能。代码已公开于https://github.com/HuiGuanLab/IMG。
原文摘要 · Abstract (English)
Video Moment Retrieval (VMR) aims to retrieve a specific moment semantically related to the given query. To tackle this task, most existing VMR methods solely focus on the visual and textual modalities while neglecting the complementary but important audio modality. Although a few recent works try to tackle the joint audio-vision-text reasoning, they treat all modalities equally and simply embed them without fine-grained interaction for moment retrieval. These designs are counter-practical as: Not all audios are helpful for video moment retrieval, and the audio of some videos may be complete noise or background sound that is meaningless to the moment determination. To this end, we propose a novel Importance-aware Multi-Granularity fusion model (IMG), which learns to dynamically and selectively aggregate the audio-vision-text contexts for VMR. Specifically, after integrating the textual guidance with vision and audio separately, we first design a pseudo-label-supervised audio importance predictor that predicts the importance score of the audio, and accordingly assigns weights to mitigate the interference caused by noisy audio. Then, we design a multi-granularity audio fusion module that adaptively fuses audio and visual modalities at local-, event-, and global-level, fully capturing their complementary contexts. We further propose a cross-modal knowledge distillation strategy to address the challenge of missing audio modality during inference. To evaluate our method, we further construct a new VMR dataset, i.e., Charades-AudioMatter, where audio-related samples are manually selected and re-organized from the original Charades-STA to validate the model's capability in utilizing audio modality. Extensive experiments validate the effectiveness of our method, achieving state-of-the-art with audio-video fusion in VMR methods. Our code is available at https://github.com/HuiGuanLab/IMG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。