让语音自然融入环境音,实现精准控制的音频生成。
VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matching

- 采用分块因果流匹配架构,支持可变时长的流式推理。
- 多奖励机制优化语音语义、准确性与时间对齐效果。
- 构建真实语音引用数据集,提升语音内容生成质量。
语音化音频合成旨在生成包含可理解语音的环境音效,应用于播客制作和视频配音等场景。现有文本到音频(T2A)系统或使语音变得模糊不可辨,或依赖独立语音合成模型后混合,导致难以控制语音出现时机及与场景的交互。本文提出VoxAudio,一种因果自回归流匹配模型,从架构、偏好与数据三方面协同解决该问题。在架构层面,采用分块因果分解与独立噪声水平设计,支持滑动窗口流式推理,并通过随机分块边界预训练实现任意粒度推理;在偏好层面,引入多奖励负向感知微调(NFT),联合优化语义保真度、语言准确性、美学质量与时间定位;在数据层面,构建大规模标注数据集VoxCorpus,其字幕直接引用嵌入语音的原文及时间区间,并建立区间标注基准VoxBench,配备时间定位评估指标。在四个涵盖通用音频、语音与统一语音化音频的基准上验证了VoxAudio的有效性与高效性。代码与演示已公开。
原文摘要 · Abstract (English)
Vocalized audio synthesis, the task of generating audio in which intelligible speech is embedded within an environmental soundscape, underpins applications such as podcast production and video dubbing. Existing Text-to-Audio (T2A) systems either reduce quoted speech to unintelligible vocal murmur or delegate it to a separate TTS model with post-hoc mixing, which forfeits control over when speech occurs and how it interacts with the scene. We present VoxAudio, a causal autoregressive flow matching model that addresses this problem from three complementary aspects. At the architecture level, chunk-wise causal factorization with independent per-chunk noise levels lets audio be emitted through sliding-window streaming inference with KV caching at variable target durations; to enable inference at arbitrary chunk granularities, we further pretrain the model with randomized chunk boundaries. At the preference level, multi-reward Negative-aware FineTuning (NFT) jointly optimizes semantic fidelity, linguistic accuracy, aesthetic quality, and temporal grounding At the data level, to supply the missing supervision for vocal content, we build VoxCorpus, a large-scale corpus whose captions quote the verbatim transcript of embedded speech with time intervals, and VoxBench, an interval-annotated benchmark with a temporal-grounding metric. Experiments on four benchmarks spanning general audio, speech, and unified vocalized audio validate the effectiveness and efficiency of VoxAudio. Our code and demos are available at https://voxaudio.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。