用小模块对齐语音与文本模型，实现高效端到端语音翻译。
Aligning Pre-trained Models for Spoken Language Translation
- 通过小连接器将冻结的语音识别与翻译模型对齐，仅优化连接器部分。
- 增大基础模型规模可显著提升翻译效果，连接器仅占模型总量5%以下。
- 连接器可充当领域适配器，有效提升跨领域语音翻译性能。
本文提出一种基于对齐冻结预训练自动语音识别（ASR）与机器翻译（MT）模型的新方法，采用小型连接模块（Q-Former，即子采样-变换器编码器）实现端到端语音翻译（ST）。该连接器桥接语音与文本模态，将ASR编码器的嵌入转换为MT编码器的潜在表示空间，且仅此部分在训练中优化。在How2英语-葡萄牙语数据集上进行实验，研究小规模场景下的对齐策略。保持连接器大小恒定且极小（小于较大模型的5%），随着基础ASR与MT模型规模和能力的提升，翻译性能普遍提高。此外发现，连接器可作为基础MT模型的领域适配器，在对齐的语音翻译设置中显著提升表现。结论表明，该方法是训练端到端语音翻译系统的可行且可扩展方案。
原文摘要 · Abstract (English)
This paper investigates a novel approach to end-to-end speech translation (ST) based on aligning frozen pre-trained automatic speech recognition (ASR) and machine translation (MT) models via a small connector module (Q-Former, our Subsampler-Transformer Encoder). This connector bridges the gap between the speech and text modalities, transforming ASR encoder embeddings into the latent representation space of the MT encoder while being the only part of the system optimized during training. Experiments are conducted on the How2 English-Portuguese dataset as we investigate the alignment approach in a small-scale scenario focusing on ST. While keeping the size of the connector module constant and small in comparison ( < 5% of the size of the larger aligned models), increasing the size and capability of the foundation ASR and MT models universally improves translation results. We also find that the connectors can serve as domain adapters for the foundation MT models, significantly improving translation performance in the aligned ST setting. We conclude that this approach represents a viable and scalable approach to training end-to-end ST systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。