通过频域分解与重构,提升音视频分割的鲁棒性
Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation
- 将音视频分割重构为频域分解与重组问题
- 在三个数据集上达到当前最优性能
- 适合关注多模态融合与噪声鲁棒性的研究者
音视频分割(AVS)在多模态机器学习中至关重要,能有效结合音频与视觉线索,精确分割视觉场景中的物体或区域。近期方法虽有显著提升,但忽视了音频与视觉模态间的固有频域矛盾——音频高频信号普遍存在干扰噪声,而视觉高频信号则包含丰富的结构细节。忽略此差异会导致性能不佳。本文从更深层视角重新思考AVS任务,将其重构为频域分解与重组问题。提出新型频率感知音视频分割框架(FAVS),包含两个关键模块:频域增强分解器(FDED)和协同跨模态一致性(SCMC)模块。FDED采用基于残差的迭代频域分解,区分模态特异性语义与结构特征;SCMC利用专家混合架构,通过动态专家路由强化语义一致性并保留模态特异性特征。大量实验表明,所提FAVS框架在三个基准数据集上达到最先进性能,丰富的定性可视化进一步验证了FDED与SCMC模块的有效性。代码将在论文接收后开源。
原文摘要 · Abstract (English)
Audio-visual segmentation (AVS) plays a critical role in multimodal machine learning by effectively integrating audio and visual cues to precisely segment objects or regions within visual scenes. Recent AVS methods have demonstrated significant improvements. However, they overlook the inherent frequency-domain contradictions between audio and visual modalities--the pervasively interfering noise in audio high-frequency signals vs. the structurally rich details in visual high-frequency signals. Ignoring these differences can result in suboptimal performance. In this paper, we rethink the AVS task from a deeper perspective by reformulating AVS task as a frequency-domain decomposition and recomposition problem. To this end, we introduce a novel Frequency-Aware Audio-Visual Segmentation (FAVS) framework consisting of two key modules: Frequency-Domain Enhanced Decomposer (FDED) module and Synergistic Cross-Modal Consistency (SCMC) module. FDED module employs a residual-based iterative frequency decomposition to discriminate modality-specific semantics and structural features, and SCMC module leverages a mixture-of-experts architecture to reinforce semantic consistency and modality-specific feature preservation through dynamic expert routing. Extensive experiments demonstrate that our FAVS framework achieves state-of-the-art performance on three benchmark datasets, and abundant qualitative visualizations further verify the effectiveness of the proposed FDED and SCMC modules. The code will be released as open source upon acceptance of the paper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。