将人脸视频转为自然语音，通过分解声学特征提升生成质量
V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow
- 分解语音为内容、音高和说话人信息三部分分别建模
- 在真实场景数据上超越现有方法，甚至优于真人语音自然度
- 适合语音合成、影视修复等需要高质量语音生成的场景
本文提出V2SFlow，一种从无声口型视频直接生成自然可懂语音的新型视频转语音框架。尽管近期V2S系统在受限数据集上表现良好，但在真实世界无约束数据上因语音信号的复杂性与多样性导致性能下降。为此，我们把语音信号分解为内容、音高和说话人信息三个可管理子空间，分别从视觉输入中预测。为生成连贯真实的语音，采用基于Transformer架构的修正流匹配解码器，从随机噪声高效建模到目标语音分布的路径。大量实验表明，V2SFlow显著优于当前最优方法，甚至在自然度上超过真实语音。代码与模型已开源：https://github.com/kaistmm/V2SFlow
原文摘要 · Abstract (English)
In this paper, we introduce V2SFlow, a novel Video-to-Speech (V2S) framework designed to generate natural and intelligible speech directly from silent talking face videos. While recent V2S systems have shown promising results on constrained datasets with limited speakers and vocabularies, their performance often degrades on real-world, unconstrained datasets due to the inherent variability and complexity of speech signals. To address these challenges, we decompose the speech signal into manageable subspaces (content, pitch, and speaker information), each representing distinct speech attributes, and predict them directly from the visual input. To generate coherent and realistic speech from these predicted attributes, we employ a rectified flow matching decoder built on a Transformer architecture, which models efficient probabilistic pathways from random noise to the target speech distribution. Extensive experiments demonstrate that V2SFlow significantly outperforms state-of-the-art methods, even surpassing the naturalness of ground truth utterances. Code and models are available at: https://github.com/kaistmm/V2SFlow
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。