统一生成语音与歌声,让模型既能自然说话又能精准控调。
UniVoice: A Unified Model for Speech and Singing Voice Generation

- 将条件分解为内容、旋律、音色,分别编码后输入共享模型
- 语音用空旋律标记,避免旋律约束;歌声用MIDI控制,实现精准节奏
- 30小时语音+35小时歌声训练,语音/歌声生成均优于现有统一模型
文本到语音(TTS)和歌唱语音合成(SVS)均旨在从符号输入生成人声音频,但对生成过程的要求不同:语音生成依赖灵活的语言驱动韵律,而歌唱生成需要显式的旋律控制和精确的节奏对齐。这种差异使得训练一个能同时生成自然语音和可控歌声的单一模型极具挑战性,因为旋律相关条件应严格约束歌唱,却不应限制语音韵律。我们提出UniVoice,一种基于条件流匹配的统一语音与歌唱生成框架。不同于使用单一不分化的条件表示,UniVoice将条件分解为内容、旋律和音色,并通过模态适配的编码器进行编码,由共享的扩散Transformer(DiT)主干网络处理。对于歌唱,旋律条件以MIDI音符序列表示;对于语音,则替换为学习得到的空旋律标记,使模型能从语言和声学上下文中推断韵律。该设计在保留歌唱显式旋律控制的同时,避免了对语音韵律施加旋律约束。我们进一步分析空旋律标记作为条件流中旋律边际化的近似。模型在30千小时语音和35千小时歌唱数据上训练,语音字错误率(PER)达5.26%,接近专用TTS系统如F5-TTS(5.21%)和CosyVoice3(5.30%)。在歌唱生成方面,PER为16.22%,优于统一基线Vevo1.5(24.72%)。
原文摘要 · Abstract (English)
Text-to-speech (TTS) and singing voice synthesis (SVS) both aim to generate human vocal audio from symbolic inputs, but they impose different requirements on the generation process. Speech generation relies on flexible, language-driven prosody, whereas singing generation requires explicit melody control and accurate rhythmic alignment. This mismatch makes it challenging to train a single model that can generate both natural speech and controllable singing, since melody-related conditions should strongly constrain singing but should not restrict speech prosody. We present UniVoice, a unified speech and singing voice generation framework based on conditional flow matching. Instead of using a single undifferentiated conditioning representation, UniVoice factorizes the condition into content, melody, and timbre, which are encoded by modality-appropriate encoders and consumed by a shared Diffusion Transformer (DiT) backbone. For singing, the melody condition is represented by MIDI note sequences; for speech, it is replaced with a learned null melody token, allowing the model to infer prosody from linguistic and acoustic context. This design preserves explicit melody control for singing while avoiding the need to impose melody constraints on speech. We further analyze the null melody token as an approximation to melody marginalization in the conditional flow. Trained on 30k hours of speech and 35k hours of singing data, UniVoice achieves a speech PER of 5.26\%, comparable to dedicated TTS systems such as F5-TTS (5.21\%) and CosyVoice3 (5.30\%). On singing generation, UniVoice achieves a PER of 16.22\%, outperforming the unified baseline Vevo1.5 (24.72\%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。