PAST通过联合建模音素与语音重建，实现无需预训练模型的端到端语音分词。
PAST: Phonetic-Acoustic Speech Tokenizer
- 利用监督音素数据，通过辅助任务直接融入领域知识进行分词
- 在音素表示和语音重建上均优于现有基线模型
- 支持实时应用，适合语音生成与语言模型的基础表示
我们提出PAST，一种新颖的端到端框架，联合建模音素信息与信号重建，无需依赖外部预训练模型。与以往依赖自监督预训练模型的方法不同，PAST采用监督音素数据，通过辅助任务将领域知识直接融入分词过程。此外，我们引入了可流式处理的因果变体PAST，支持实时语音应用。结果表明，PAST在常见评估指标上超越了现有基线分词器，包括音素表示与语音重建性能。值得注意的是，当作为语音语言模型的语音表示时，PAST也表现出更优性能，进一步凸显其在口语生成中的基础价值。为促进后续研究，我们公开了完整实现。代码、模型检查点及样本见：https://pages.cs.huji.ac.il/adiyoss-lab/PAST
原文摘要 · Abstract (English)
We present PAST, a novel end-to-end framework that jointly models phonetic information alongside signal reconstruction, eliminating the need for external pretrained models. Unlike previous approaches that rely on pretrained self-supervised models, PAST employs supervised phonetic data, directly integrating domain knowledge into the tokenization process via auxiliary tasks. Additionally, we introduce a streamable, causal variant of PAST, enabling real-time speech applications. Results demonstrate that PAST surpasses existing evaluated baseline tokenizers across common evaluation metrics, including phonetic representation and speech reconstruction. Notably, PAST also achieves superior performance when serving as a speech representation for speech language models, further highlighting its effectiveness as a foundation for spoken language generation. To foster further research, we release the full implementation. For code, model checkpoints, and samples see: https://pages.cs.huji.ac.il/adiyoss-lab/PAST
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。