Metis用自监督预训练统一生成语音，少样本也能超好用。
Metis: A Foundation Speech Generation Model with Masked Generative Pre-training
- 用掩码生成预训练，基于30万小时无标注语音数据。
- 仅用不到2000万参数，在5项任务上超越现有系统。
- 支持多模态输入，小数据下仍能高效适配新任务。
我们提出Metis，一种用于统一语音生成的基础模型。与以往特定任务或多任务模型不同，Metis采用预训练-微调范式：在大规模无标签语音数据上通过掩码生成建模进行预训练，再通过任务特定条件微调以适应多样语音生成任务。具体而言，Metis采用两种离散语音表示：源自语音自监督学习（SSL）特征的SSL token，以及直接从波形量化得到的声学token；在不依赖额外条件的情况下，利用30万小时多样化语音数据对SSL token进行掩码生成预训练；通过微调引入任务条件，实现对多种语音生成任务的高效适应，支持多模态输入，即使在参数量少于2000万或训练数据仅为以往1/300的情况下依然表现优异。实验表明，Metis在零样本文本到语音、语音转换、目标说话人提取、语音增强和唇语到语音等五项任务中均超越现有最优任务专用或多任务系统。
原文摘要 · Abstract (English)
We introduce Metis, a foundation model for unified speech generation. Unlike previous task-specific or multi-task models, Metis follows a pre-training and fine-tuning paradigm. It is pre-trained on large-scale unlabeled speech data using masked generative modeling and then fine-tuned to adapt to diverse speech generation tasks. Specifically, 1) Metis utilizes two discrete speech representations: SSL tokens derived from speech self-supervised learning (SSL) features, and acoustic tokens directly quantized from waveforms. 2) Metis performs masked generative pre-training on SSL tokens, utilizing 300K hours of diverse speech data, without any additional condition. 3) Through fine-tuning with task-specific conditions, Metis achieves efficient adaptation to various speech generation tasks while supporting multimodal input, even when using limited data and trainable parameters. Experiments demonstrate that Metis can serve as a foundation model for unified speech generation: Metis outperforms state-of-the-art task-specific or multi-task systems across five speech generation tasks, including zero-shot text-to-speech, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech, even with fewer than 20M trainable parameters or 300 times less training data. Audio samples are are available at https://metis-demo.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。