用自监督目标蒸馏HuBERT，模型更小效果更好
DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective
- 用HuBERT的自监督目标直接训练学生模型，无需额外模块
- 在SUPERB上语音识别提升超14%，音素识别提升超21%
- 适合需要轻量化语音模型的部署场景
我们提出DiceHuBERT，一种用于压缩广泛使用的自监督学习语音基础模型HuBERT的知识蒸馏框架。与依赖教师-学生模型层间、特征间映射的现有方法不同，DiceHuBERT通过直接用学生模型替换原模型，利用HuBERT的迭代自蒸馏机制，使学生模型在与预训练时相同的自监督学习目标下进行训练，无需额外模块或架构约束。在SUPERB数据集上的实验表明，DiceHuBERT持续优于现有蒸馏方法，音素识别性能提升超过21%，自动语音识别性能提升超过14%。此外，它在多个任务上表现出色，展现出明显优势。
原文摘要 · Abstract (English)
We introduce DiceHuBERT, a knowledge distillation framework for compressing HuBERT, a widely used self-supervised learning (SSL)-based speech foundation model. Unlike existing distillation methods that rely on layer-wise and feature-wise mapping between teacher and student models, DiceHuBERT leverages HuBERT's iterative self-distillation mechanism by directly replacing the original model with a student model. This replacement allows the student to be trained using the same SSL objective used when pre-training HuBERT, eliminating the need for additional modules or architectural constraints. Experimental results on SUPERB show that DiceHuBERT consistently outperforms existing distillation methods, improving phoneme recognition performance by over 21% and ASR performance by more than 14%. Furthermore, DiceHuBERT demonstrates competitive performance across multiple tasks, highlighting its clear advantage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。