仅用1%的模拟数据，让大模型直接实现边听边译。
Direct Simultaneous Translation Activation for Large Audio-Language Models
- 通过随机截断语音生成同步翻译数据，无需改架构
- 仅用1%同步数据就激活模型实时翻译能力
- 适合想快速部署实时翻译的大模型应用者
同时语音到文本翻译（Simul-S2TT）旨在实时将语音翻译为目标文本，在接收源语音输入的同时输出翻译结果，而非等待完整语句说完。以往研究常通过修改模型结构来实现读写策略。然而随着大音频-语言模型（LALMs）的发展，关键挑战在于如何在不增加额外结构改动的前提下，直接激活其同步翻译能力。本文提出同步自增强（SimulSA）策略，利用LALMs自身能力，通过随机截断语音并构建部分对齐的翻译数据，生成同步训练样本。将其融入离线SFT数据中，有效弥合预训练阶段离线翻译与推理阶段同步翻译之间的分布差异。实验表明，仅需在完整离线SFT数据基础上，额外加入约1%的同步数据，即可显著激活LALMs的Simul-S2TT能力，且无需修改模型架构或解码策略。
原文摘要 · Abstract (English)
Simultaneous speech-to-text translation (Simul-S2TT) aims to translate speech into target text in real time, outputting translations while receiving source speech input, rather than waiting for the entire utterance to be spoken. Simul-S2TT research often modifies model architectures to implement read-write strategies. However, with the rise of large audio-language models (LALMs), a key challenge is how to directly activate Simul-S2TT capabilities in base models without additional architectural changes. In this paper, we introduce {\bf Simul}taneous {\bf S}elf-{\bf A}ugmentation ({\bf SimulSA}), a strategy that utilizes LALMs' inherent capabilities to obtain simultaneous data by randomly truncating speech and constructing partially aligned translation. By incorporating them into offline SFT data, SimulSA effectively bridges the distribution gap between offline translation during pretraining and simultaneous translation during inference. Experimental results demonstrate that augmenting only about {\bf 1\%} of the simultaneous data, compared to the full offline SFT data, can significantly activate LALMs' Simul-S2TT capabilities without modifications to model architecture or decoding strategy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。