不靠预训练也能高效实现语音直接翻译，只需微调解码器注意力机制。
Unveiling the Role of Pretraining in Direct Speech Translation
- 在解码器注意力中引入早期输入信息，帮助从零开始训练的模型更好理解语音。
- 无需预训练的模型性能接近预训练版本，且训练时间显著缩短。
- 适合想节省资源、快速部署语音翻译系统的研究者与开发者。
直接语音到文本翻译系统面临数据稀缺问题。传统方法通常在自动语音识别任务上预训练编码器，但会降低训练效率。本研究对比了使用预训练编码器和从零开始训练的系统训练动态。结果发现，随机初始化模型在整个训练过程中难以有效利用语音输入信息进行预测。我们推测，这是由于直接语音翻译的编码器训练难度大：从零开始需同时学习声学与语义建模，而预训练模型可专注语义。基于此，我们提出在解码器交叉注意力中加入早期训练步骤的源信息。实验表明，经此改进后，从零训练的模型性能可媲美预训练模型，且训练时间更短。
原文摘要 · Abstract (English)
Direct speech-to-text translation systems encounter an important drawback in data scarcity. A common solution consists on pretraining the encoder on automatic speech recognition, hence losing efficiency in the training process. In this study, we compare the training dynamics of a system using a pretrained encoder, the conventional approach, and one trained from scratch. We observe that, throughout the training, the randomly initialized model struggles to incorporate information from the speech inputs for its predictions. Hence, we hypothesize that this issue stems from the difficulty of effectively training an encoder for direct speech translation. While a model trained from scratch needs to learn acoustic and semantic modeling simultaneously, a pretrained one can just focus on the latter. Based on these findings, we propose a subtle change in the decoder cross-attention to integrate source information from earlier steps in training. We show that with this change, the model trained from scratch can achieve comparable performance to the pretrained one, while reducing the training time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。