让语音听起来像在不同场景录制的，仅用无标签视频训练。
Self-Supervised Audio-Visual Soundscape Stylization
- 用自监督学习，从视频中提取音视频片段做条件提示。
- 能成功将目标场景的声音特性迁移到输入语音上。
- 视觉信息能提升声音预测效果，适合音频风格迁移研究者。
语音蕴含大量场景信息，如混响和背景声等。本文提出一种方法，将输入语音处理为仿佛在另一场景中录制的效果，只需一个来自该场景的音视频条件样本。模型通过自监督学习，利用自然视频中重复出现的声音事件和纹理。具体地，从视频中提取音频片段并进行语音增强，随后训练潜空间扩散模型以恢复原始语音，使用视频中另一片段作为条件提示。通过此过程，模型学会将条件样本的声音属性迁移到输入语音。实验表明，该模型可使用未标注的野外视频成功训练，且额外的视觉信号能提升声音预测性能。
原文摘要 · Abstract (English)
Speech sounds convey a great deal of information about the scenes, resulting in a variety of effects ranging from reverberation to additional ambient sounds. In this paper, we manipulate input speech to sound as though it was recorded within a different scene, given an audio-visual conditional example recorded from that scene. Our model learns through self-supervision, taking advantage of the fact that natural video contains recurring sound events and textures. We extract an audio clip from a video and apply speech enhancement. We then train a latent diffusion model to recover the original speech, using another audio-visual clip taken from elsewhere in the video as a conditional hint. Through this process, the model learns to transfer the conditional example's sound properties to the input speech. We show that our model can be successfully trained using unlabeled, in-the-wild videos, and that an additional visual signal can improve its sound prediction abilities. Please see our project webpage for video results: https://tinglok.netlify.app/files/avsoundscape/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。