提出新模型提升音视频分割精度，兼顾局部全局信息与跨模态对齐。
Complementary and Contrastive Learning for Audio-Visual Segmentation
- 用并行双路结构融合多尺度视觉与音频特征，增强跨模态互补性。
- 设计多查询注意力模块，同时捕捉帧内空间与视频级时序关系。
- 引入双模态对比学习，在统一特征空间对齐音视频语义，适合多模态研究者。
音视频分割（AVS）旨在生成与物体声音信号对应的像素级分割图。尽管已有众多基于CNN和Transformer的方法显著提升了分割准确率与鲁棒性，传统CNN方法受限于局部感受野，仅通过拼接、乘法等基础操作处理音视频交互；近期的Transformer方法虽以音频为查询利用注意力机制增强帧内跨模态协作，但仍难以充分提取多模态系数与时间动态。为此，本文提出互补与对比学习框架CCFormer，能够全面捕捉时空上下文。该框架首先通过早期融合模块（EIM）采用并行双路结构，将多尺度视觉特征与音频数据融合，强化跨模态互补性；随后引入多查询变压器模块（MTM），动态赋予音频查询学习能力，同步建模帧内空间特征与视频级关联关系；此外，提出双模态对比学习（BCL），促进两模态在统一特征空间中的对齐。通过上述设计的有效结合，本方法在S4、MS3和AVSS数据集上均达到新的最先进水平。代码与模型权重将公开发布于https://github.com/SitongGong/CCFormer。
原文摘要 · Abstract (English)
Audio-Visual Segmentation (AVS) aims to generate pixel-wise segmentation maps that correlate with the auditory signals of objects. This field has seen significant progress with numerous CNN and Transformer-based methods enhancing the segmentation accuracy and robustness. Traditional CNN approaches manage audio-visual interactions through basic operations like padding and multiplications but are restricted by CNNs' limited local receptive field. More recently, Transformer-based methods treat auditory cues as queries, utilizing attention mechanisms to enhance audio-visual cooperation within frames. Nevertheless, they typically struggle to extract multimodal coefficients and temporal dynamics adequately. To overcome these limitations, we present the Complementary and Contrastive Transformer (CCFormer), a novel framework adept at processing both local and global information and capturing spatial-temporal context comprehensively. Our CCFormer initiates with the Early Integration Module (EIM) that employs a parallel bilateral architecture, merging multi-scale visual features with audio data to boost cross-modal complementarity. To extract the intra-frame spatial features and facilitate the perception of temporal coherence, we introduce the Multi-query Transformer Module (MTM), which dynamically endows audio queries with learning capabilities and models the frame and video-level relations simultaneously. Furthermore, we propose the Bi-modal Contrastive Learning (BCL) to promote the alignment across both modalities in the unified feature space. Through the effective combination of those designs, our method sets new state-of-the-art benchmarks across the S4, MS3 and AVSS datasets. Our source code and model weights will be made publicly available at https://github.com/SitongGong/CCFormer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。