arXiv:2608.06424cs.SDcs.CL2026-08

用分层编码器扩散模型实现精准语音修复与编辑,保持音色和语调一致。

Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing

论文配图:Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing
图 1 · 摘自论文原文
  • 基于分层编码器的扩散框架,逐层修复语音缺失部分。
  • 在RealEdit基准上超越现有方法,修复和编辑效果均最优。
  • 适合需要保持说话人特征的语音修复与内容修改任务。

语音记录常存在缺失、损坏或错误区域,需在不重新生成整个语音的前提下进行重建或修改。语音修复用于恢复缺失片段,语音编辑则根据修改后的文本替换原有内容。两者均要求生成语音准确表达目标词语,同时保持说话人特征、语调、节奏和录音条件的一致性。离散扩散模型因其可迭代精修掩码标记并联合依赖左右声学上下文,特别适合此类任务。我们提出SIEDD,一种基于分层编码器令牌的文本引导语音修复与编辑离散扩散框架。其核心架构HiCoDD遵循RVQ生成顺序,将已生成的码本作为干净的承诺声学上下文,仅对当前待优化码本施加扩散。这种分离设计实现无泄漏联合训练,并匹配从粗到细的推理顺序。模型还结合了音素级条件控制、局部跨度的无分类器指导和时长预测,支持固定时长修复与可变时长文本编辑。在RealEdit基准上,SIEDD在所有评估方法中取得最佳整体编辑性能;在单个与多个缺口的语音修复设置下,也全面优于对比的自回归基线。结果表明,显式建模编码器层次结构显著提升上下文保持的语音重建与编辑能力。

原文摘要 · Abstract (English)

Speech recordings often contain missing, corrupted, or incorrect regions that must be reconstructed or modified without re-synthesizing the entire utterance. Speech inpainting restores missing segments, whereas speech editing replaces spoken content according to an edited transcript. Both tasks require the generated speech to express the intended words while remaining consistent with the surrounding speaker identity, prosody, timing, and recording conditions. Discrete diffusion is particularly well suited to these tasks because it can iteratively refine masked tokens while jointly conditioning on both left and right acoustic context. We introduce SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing over hierarchical codec tokens. Its core architecture, HiCoDD, follows the RVQ generation order by representing previously generated codebooks as clean, committed acoustic context and applying diffusion only to the current refinement codebook. This separation enables leakage-free joint training while matching sequential coarse-to-fine inference. The model further combines phoneme-level conditioning, span-localized classifier-free guidance, and duration prediction to support both fixed-duration inpainting and variable-duration text edits. On the RealEdit benchmark, SIEDD achieves the best overall speech-editing performance among the evaluated methods. It also outperforms the evaluated autoregressive baselines across all speech-inpainting settings, on both single and multiple gaps. These results demonstrate that explicitly modeling the codec hierarchy substantially improves context-preserving speech reconstruction and editing. See our full code at https://github.com/iftachShoham/SIEDD.

语音修复扩散模型文本编辑

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。