用视觉信息生成更精准的立体声，提升空间感与细节。
CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation
- 引入视听条件归一化层，动态对齐音频特征
- 通过对比学习增强空间敏感性，提升细节表现
- 测试时增强策略提升性能，适合音视频融合应用
双耳音频生成（BAG）旨在利用视觉提示将单声道音频转换为立体声，需深入理解空间与语义信息。现有模型易过度依赖房间环境，丢失精细空间细节。本文提出新模型，引入视听条件归一化层，动态利用视觉上下文对齐目标差异音频特征的均值与方差；设计一种新对比学习方法，从打乱的视觉特征中挖掘负样本以增强空间敏感性；并提出一种高效测试时增强策略，用于视频数据提升性能。在FAIR-Play和MUSIC-Stereo基准上取得当前最优生成精度。
原文摘要 · Abstract (English)
Binaural audio generation (BAG) aims to convert monaural audio to stereo audio using visual prompts, requiring a deep understanding of spatial and semantic information. However, current models risk overfitting to room environments and lose fine-grained spatial details. In this paper, we propose a new audio-visual binaural generation model incorporating an audio-visual conditional normalisation layer that dynamically aligns the mean and variance of the target difference audio features using visual context, along with a new contrastive learning method to enhance spatial sensitivity by mining negative samples from shuffled visual features. We also introduce a cost-efficient way to utilise test-time augmentation in video data to enhance performance. Our approach achieves state-of-the-art generation accuracy on the FAIR-Play and MUSIC-Stereo benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。