arXiv:2504.11809cs.CL2025-04被引 13

提出全单向架构的实时语音翻译模型,提升效率与适应性。

Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture

  • 采用全单向结构,避免重复编码,降低计算开销。
  • 多延迟数据构建与动态读写策略,实现更低延迟与更高准确率。
  • 适合需要低延迟实时翻译的场景,如会议同传、直播字幕。

同时语音翻译(SimulST)在处理部分语音输入时逐步生成翻译结果。尽管大语言模型(LLM)在离线翻译任务中表现优异,但将其应用于SimulST仍面临挑战:现有基于LLM的方法或因双向语音编码重复计算导致高开销,或依赖固定读写策略,限制效率与性能。本文提出高效自适应同时语音翻译模型EASiST,采用全单向架构,包含语音编码器与LLM。EASiST引入多延迟数据构建策略,生成语义对齐的训练样本,并将SimulST重新定义为带有显式读写标记的交错生成任务。为支持自适应推理,设计轻量级策略头以动态预测读写动作。此外,采用多阶段训练策略对齐语音-文本模态,优化翻译与策略行为。在MuST-C En→De和En→Es数据集上的实验表明,相比多个强基线,EASiST在延迟与质量权衡上表现更优。

原文摘要 · Abstract (English)

Simultaneous speech translation (SimulST) produces translations incrementally while processing partial speech input. Although large language models (LLMs) have showcased strong capabilities in offline translation tasks, applying them to SimulST poses notable challenges. Existing LLM-based SimulST approaches either incur significant computational overhead due to repeated encoding of bidirectional speech encoder, or they depend on a fixed read/write policy, limiting the efficiency and performance. In this work, we introduce Efficient and Adaptive Simultaneous Speech Translation (EASiST) with fully unidirectional architecture, including both speech encoder and LLM. EASiST includes a multi-latency data curation strategy to generate semantically aligned SimulST training samples and redefines SimulST as an interleaved generation task with explicit read/write tokens. To facilitate adaptive inference, we incorporate a lightweight policy head that dynamically predicts read/write actions. Additionally, we employ a multi-stage training strategy to align speech-text modalities and optimize both translation and policy behavior. Experiments on the MuST-C En$\rightarrow$De and En$\rightarrow$Es datasets demonstrate that EASiST offers superior latency-quality trade-offs compared to several strong baselines.

语音翻译实时翻译大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。