动态调整语音模型各层特征权重，提升下游任务表现。
VARAN: Variational Inference for Self-Supervised Speech Models Fine-Tuning on Downstream Tasks
- 根据输入动态分配各层特征重要性，避免固定加权缺陷。
- 在语音识别与情感识别任务中显著优于传统方法。
- 特别适合结合LoRA微调，适用于高效语音模型适配场景。
传统自监督语音模型微调时的层聚合方法（如使用最后一层或加权求和）存在信息瓶颈和对所有样本采用静态特征权重的问题。本文提出VARAN框架，通过层专用探测头和数据相关加权机制，动态为每个输入调整层特征优先级。在自动语音识别与语音情感识别任务上的实验表明，该方法性能优越，尤其在使用LoRA微调技术时表现更佳。该框架解决了保留层特异性信息与灵活利用特征之间的权衡，推动了自监督语音表示的高效适配。
原文摘要 · Abstract (English)
Conventional methods for aggregating layers in fine-tuned self-supervised speech models, such as using the final layer or weighted sum, suffer from information bottlenecks and static feature weighting for all dataset examples. We propose VARAN, a framework that dynamically tailors layer aggregation to individual inputs. By employing layer-specialized probing heads and data-dependent weighting, VARAN adaptively prioritizes layer's features based on input. Evaluations on automatic speech recognition and speech emotion recognition tasks demonstrate VARAN's superior performance, particularly when using the LoRA fine-tuning technique. The framework resolves the trade-off between preserving layer-specific information and enabling flexible feature utilization, advancing efficient adaptation of self-supervised speech representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。