通过分离音视频对齐与重建任务,提升跨模态表示学习效果。
Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning
- 分路径优化:对比学习与重建分开进行,避免干扰
- 零样本检索性能提升:音频到视频检索达37.1%
- 适合做大规模音视频预训练的模型设计
近期音视频表示学习进展表明,联合使用对比对齐与掩码重建具有价值。然而,在单次前向传播中同时优化这两个目标,会导致对比分支依赖为重建设计的随机可见补丁,而非跨模态对齐,引入语义噪声和优化干扰。本文提出TG-DP框架,将重建与对齐解耦至独立优化路径。通过分离两分支的掩码策略,使对比路径可采用更适合跨模态对齐的可见性模式。教师模型进一步为该路径提供辅助指导,组织可见标记,减少干扰并稳定表示学习。TG-DP在零样本检索上达到当前最优表现:在AudioSet上,视频到音频检索的R@1从35.2%提升至37.4%,音频到视频检索从27.9%提升至37.1%。所学表示亦具语义鲁棒性,在AS20K和VGGSound上线性探测性能达领先水平。结果表明,解耦多模态目标并引入教师引导结构,是提升大规模音视频预训练的有效方案。代码已公开于https://github.com/wanglg20/TG-DP。
原文摘要 · Abstract (English)
Recent advances in audio-visual representation learning have shown the value of combining contrastive alignment with masked reconstruction. However, jointly optimizing these objectives in a single forward pass forces the contrastive branch to rely on randomly visible patches designed for reconstruction rather than cross-modal alignment, introducing semantic noise and optimization interference. We propose TG-DP, a Teacher-Guided Dual-Path framework that decouples reconstruction and alignment into separate optimization paths. By disentangling the masking regimes of the two branches, TG-DP enables the contrastive pathway to use a visibility pattern better suited to cross-modal alignment. A teacher model further provides auxiliary guidance for organizing visible tokens in this branch, helping reduce interference and stabilize cross-modal representation learning. TG-DP achieves state-of-the-art performance in zero-shot retrieval. On AudioSet, it improves R@1 from 35.2\% to 37.4\% for video-to-audio retrieval and from 27.9\% to 37.1\% for audio-to-video retrieval. The learned representations also remain semantically robust, achieving state-of-the-art linear-probe performance on AS20K and VGGSound. Taken together, our results suggest that decoupling multimodal objectives and introducing teacher-guided structure into the contrastive pathway provide an effective framework for improving large-scale audio-visual pretraining. Code is available at https://github.com/wanglg20/TG-DP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。