arXiv:2412.08237cs.SDcs.CL2024-12被引 13

用简单方法提升语音合成数据质量,降低训练和部署成本。

TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch

  • 用噪声鲁棒的音频分词器简化数据处理流程,保留超50%原始数据。
  • 将语音生成模块替换为大模型架构,实现端到端推理与低部署成本。
  • 首次探索语音合成与识别共用数据集训练的可行性,适合落地应用者。

众所周知,基于大模型的语音合成系统对数据需求量大。现有工作通常采用复杂的数据处理流水线来获取高质量训练数据,这些流程需多个高精度模型(如语音降噪、增强、说话人分离、标点预测),而这些模型本身又依赖高质量数据,且极少开源。即使使用先进模型,仍存在背景噪声未完全清除、标点与实际停顿错位等问题。此外,严格的过滤策略常导致仅保留10%-30%的原始数据,严重阻碍数据扩展。本文利用噪声鲁棒的音频分词器(S3Tokenizer),设计了一套简化但高效的数据处理流程,在保持数据质量的同时,将数据保留率提升至50%以上。此外,基于大模型的语音合成系统部署成本较高,当前系统通常仅用大模型生成文本-分词序列,而依赖独立模型(如流匹配模型)完成分词到波形生成,无法直接在大模型推理引擎上运行,进一步增加部署复杂性。为此,我们移除大模型与流模型中的冗余模块,将流模型主干替换为大模型架构。在此基础上,提出统一的流式与非流式推理架构,显著降低部署成本。最后,得益于简化流程与S3Tokenizer降低的训练数据质量要求,我们探索了使用同一组数据同时训练语音合成与语音识别任务的可行性。

原文摘要 · Abstract (English)

It is well known that LLM-based systems are data-hungry. Recent LLM-based TTS works typically employ complex data processing pipelines to obtain high-quality training data. These sophisticated pipelines require excellent models at each stage (e.g., speech denoising, speech enhancement, speaker diarization, and punctuation models), which themselves demand high-quality training data and are rarely open-sourced. Even with state-of-the-art models, issues persist, such as incomplete background noise removal and misalignment between punctuation and actual speech pauses. Moreover, the stringent filtering strategies often retain only 10-30\% of the original data, significantly impeding data scaling efforts. In this work, we leverage a noise-robust audio tokenizer (S3Tokenizer) to design a simplified yet effective TTS data processing pipeline that maintains data quality while substantially reducing data acquisition costs, achieving a data retention rate of over 50\%. Beyond data scaling challenges, LLM-based TTS systems also incur higher deployment costs compared to conventional approaches. Current systems typically use LLMs solely for text-to-token generation, while requiring separate models (e.g., flow matching models) for token-to-waveform generation, which cannot be directly executed by LLM inference engines, further complicating deployment. To address these challenges, we eliminate redundant modules in both LLM and flow components, replacing the flow model backbone with an LLM architecture. Building upon this simplified flow backbone, we propose a unified architecture for both streaming and non-streaming inference, significantly reducing deployment costs. Finally, we explore the feasibility of unifying TTS and ASR tasks using the same data for training, thanks to the simplified pipeline and the S3Tokenizer that reduces the quality requirements for TTS training data.

语音合成大模型数据处理部署优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。