用分层离散编码建模语音,让视频生成更自然的说话声。
Hierarchical Codec Diffusion for Video-to-Speech Generation

- 分层生成语音:低层抓说话人特征,高层调语调表情。
- 在VTS任务上音质和表现力优于现有方法。
- 适合关注语音细节生成与跨模态对齐的研究者。
视频转语音(VTS)旨在从无声视频中合成语音,但现有方法忽视了语音的分层特性——从粗粒度的说话人相关语义到细粒度的语调细节。这种忽略导致视觉与语音特征在特定层次上难以直接对齐。本文基于残差向量量化(RVQ)的分层结构,提出HiCoDiT,一种分层离散扩散变换器,利用离散语音标记的内在层次性实现强音频-视觉对齐。具体地,低层标记编码粗粒度说话人语义,高层标记捕捉细粒度语调;低层块结合口型运动和面部身份生成内容,高层块通过面部表情调节语调动态。此外,提出双尺度自适应实例归一化,同时通过通道归一化捕获全局嗓音风格、通过时间归一化捕捉局部语调变化。大量实验表明,HiCoDiT在保真度与表现力上均优于基线,凸显了离散建模在VTS中的潜力。代码与语音演示见https://github.com/Jiaxin-Ye/HiCoDiT。
原文摘要 · Abstract (English)
Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals. However, existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details. This oversight hinders direct alignment between visual and speech features at specific hierarchical levels during property matching. In this paper, leveraging the hierarchical structure of Residual Vector Quantization (RVQ)-based codec, we propose HiCoDiT, a novel Hierarchical Codec Diffusion Transformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment. Specifically, since lower-level tokens encode coarse speaker-aware semantics and higher-level tokens capture fine-grained prosody, HiCoDiT employs low-level and high-level blocks to generate tokens at different levels. The low-level blocks condition on lip-synchronized motion and facial identity to capture speaker-aware content, while the high-level blocks use facial expression to modulate prosodic dynamics. Finally, to enable more effective coarse-to-fine conditioning, we propose a dual-scale adaptive instance layer normalization that jointly captures global vocal style through channel-wise normalization and local prosody dynamics through temporal-wise normalization. Extensive experiments demonstrate that HiCoDiT outperforms baselines in fidelity and expressiveness, highlighting the potential of discrete modelling for VTS. The code and speech demo are both available at https://github.com/Jiaxin-Ye/HiCoDiT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。