分两阶段生成语音，提升目标说话人提取的清晰度与保真度。
GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model
- 先生成语义标记，再生成精细声学标记，分离语义与声学提升稳定性。
- 在Libri2Mix上优于现有方法，语音质量、可懂度和说话人一致性均更高。
- 采用冻结语言模型条件训练，减少暴露偏差，适合语音增强与分离任务。
基于语言模型（LM）的生成建模已成为目标说话人提取（TSE）的有前景方向，具备更好的泛化能力和高保真语音生成潜力。本文提出GenTSE，一种两阶段的仅解码器生成式语言模型：第一阶段预测粗粒度语义标记，第二阶段生成细粒度声学标记。通过分离语义与声学表示，提升了解码稳定性并获得更准确的目标语音。两个阶段均使用连续的自监督学习（SSL）或编解码器嵌入，提供比离散提示方法更丰富的上下文信息。为降低暴露偏差，采用冻结语言模型条件（Frozen-LM Conditioning）训练策略，使语言模型在训练中基于早期检查点的预测标记进行条件生成，缩小教师强制训练与自回归推理之间的差距。进一步应用直接偏好优化（DPO），使输出更符合听觉感知偏好。在Libri2Mix数据集上的实验表明，GenTSE在语音质量、可懂度和说话人一致性方面均超越此前基于语言模型的系统。
原文摘要 · Abstract (English)
Language Model (LM)-based generative modeling has emerged as a promising direction for TSE, offering potential for improved generalization and high-fidelity speech. We propose GenTSE, a two-stage decoder-only generative LM for TSE: Stage-1 predicts coarse semantic tokens, and Stage-2 generates fine acoustic tokens. Separating semantics and acoustics stabilizes decoding and yields more accurate target speech. Both stages use continuous SSL or codec embeddings, offering richer context than discretized-prompt methods. To reduce exposure bias, we employ a Frozen-LM Conditioning training strategy that conditions the LMs on predicted tokens from earlier checkpoints to reduce the gap between teacher-forcing training and autoregressive inference. We further apply DPO to better align outputs with perceptual preferences. Experiments on Libri2Mix show that GenTSE surpasses previous LM-based systems in speech quality, intelligibility, and speaker consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。