让语音模型持续学习新任务，无需重新训练就能适应新语音数据。
Continual Speech Learning with Fused Speech Features
- 在编码器上加可学习门控融合层，动态选择任务相关特征。
- 在6个语音任务中显著提升准确率，实现零样本迁移。
- 适合需要快速适配新语音场景的实时应用开发者。
语音数据的快速增长要求模型具备自适应能力，传统静态方法难以应对动态多样的语音信息。我们提出连续语音学习新范式，旨在弥补现有语音模型在适应性上的差距。采用编码器-解码器结构的Whisper模型，将各类语音任务统一为生成式格式。在编码器顶部引入可学习的门控融合层，动态选择下游任务所需的特定特征。该方法在六个语音处理任务中显著优于传统方法，实现了无需完整重训练即可适应新语音任务的性能提升。
原文摘要 · Abstract (English)
Rapid growth in speech data demands adaptive models, as traditional static methods fail to keep pace with dynamic and diverse speech information. We introduce continuous speech learning, a new set-up targeting at bridging the adaptation gap in current speech models. We use the encoder-decoder Whisper model to standardize speech tasks into a generative format. We integrate a learnable gated-fusion layer on the top of the encoder to dynamically select task-specific features for downstream tasks. Our approach improves accuracy significantly over traditional methods in six speech processing tasks, demonstrating gains in adapting to new speech tasks without full retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。