用知识蒸馏压缩大模型，实现高效语音生成。
Efficient Interleaved Speech Modeling through Knowledge Distillation
- 通过层对齐蒸馏，匹配隐藏状态、注意力图和软标签来压缩模型。
- 3倍压缩后，性能仅下降1.4个归一化困惑度点，准确率超基线。
- 适合部署在普通硬件上，适用于实时对话系统等场景。
当前语音语言模型规模过大，难以满足多数部署环境的大小与延迟要求。本文通过层对齐的知识蒸馏，匹配隐藏状态、注意力图和软化输出，将大型多模态变压器模型压缩3倍，性能损失极小。提出TinyWave系列20亿参数模型，用于语音到语音及交错语音-文本生成，基于5万小时公开音频训练。支持（i）使用音素或表现性标记的纯语音生成，（ii）语音-文本混合续写。在Libri-Light上的评估显示，TinyWave的归一化困惑度仅比教师模型高1.4点；在语音StoryCloze和SALMon任务上，准确率达到教师模型的93%–97%，优于同规模基线。这些模型专为通用硬件部署优化，可应用于实时对话代理、辅助技术及低资源环境。我们发布模型、训练代码与评估脚本，支持紧凑且富有表现力的语音生成研究复现。
原文摘要 · Abstract (English)
Current speech language models exceed the size and latency constraints of many deployment environments. We build compact, expressive speech generation models through layer-aligned distillation, matching hidden states, attention maps, and softened logits to compress large multimodal transformers by 3x with minimal loss in performance. We introduce TinyWave, a family of 2B-parameter models for speech-to-speech and interleaved speech-text generation, trained on 50,000 hours of public audio. TinyWave supports (i) speech-only generation using phonetic or expressive tokens and (ii) mixed speech-text continuations. Evaluation on Libri-Light shows TinyWave within 1.4 normalized perplexity points of its teacher. Accuracy on spoken StoryCloze and SALMon reaches 93-97% of the teacher's performance, outperforming size-matched baselines. These models are optimized for deployment on commodity hardware, enabling applications in real-time conversational agents, assistive technologies, and low-resource environments. We release models, training code, and evaluation scripts to support reproducible research on compact, expressive speech generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。