arXiv:2506.11160eess.AScs.SD2025-06ACL被引 1

S2ST-Omni通过分层语言感知设计,实现多语种语音翻译的高精度与灵活部署。

S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translation

  • 分层架构:先语音转文本,再文本转语音,支持独立优化。
  • 双层语言感知:在声学与语义层注入源语言信息,提升跨语言一致性。
  • 可插拔合成模块:无需重训练前端,适配任意先进语音合成器。

尽管语音到语音翻译(S2ST)取得进展,但高精度与实用灵活性仍难以兼顾。本文提出S2ST-Omni,一种组合式S2ST框架,将高精度语音转文本(S2TT)前端与模块化、即插即用的文本转语音(TTS)后端结合,实现翻译与合成的独立优化。S2TT端采用混合适配器,遵循‘局部-全局’策略,连接预训练Whisper编码器与Qwen3大模型,构建分层声学到语义抽象。在此基础上,提出分层语言感知架构,在声学层通过语言感知双CTC与FiLM风格调制学习语言特异性但内容忠实的声学表示;在语言层通过动态构建源语言条件提示,激活大模型中的语言特定翻译知识。为高效优化,设计任务专用渐进式微调策略,先稳定语音-文本对齐,再基于此基础使用LoRA提升翻译性能。TTS后端完全模块化,可直接接入任意先进合成器而无需重训练前端。在CVSS-C数据集上,S2ST-Omni在法语、德语、西班牙语到英语方向均持续达到最优的BLEU与ASR-BLEU分数,超越多个近期强基线模型。

原文摘要 · Abstract (English)

Despite recent advances in speech-to-speech translation (S2ST), it remains difficult to achieve both high translation accuracy and practical flexibility. In this paper, we present S2ST-Omni, a compositional S2ST framework that integrates a high-accuracy speech-to-text translation (S2TT) frontend with a modular, plug-and-play text-to-speech (TTS) backend, enabling independent optimization of translation and synthesis. On the S2TT side, we introduce a hybrid adapter that follows a "local-then-global" strategy to bridge a pretrained Whisper encoder and a Qwen3 LLM, yielding a hierarchical acoustic-to-semantic abstraction. Building on this bridge, we further propose a hierarchical language-aware architecture that injects source-language information at two complementary levels. At the acoustic level, Language-Aware Dual-CTC operates on intermediate adapter features and employs FiLM-style feature modulation with a learnable gate, encouraging the model to learn language-specific but content-faithful acoustic representations. At the linguistic level, Language-Aware Prompting dynamically constructs source-language-conditioned prompts that activate language-specific translation knowledge in the LLM. To enable efficient optimization, we design a task-specific progressive fine-tuning strategy that first stabilizes speech-text alignment and then improves translation via LoRA on top of this converged foundation. The TTS backend remains fully modular and can be instantiated with any state-of-the-art synthesizer without retraining the S2TT frontend. Experiments on CVSS-C show that S2ST-Omni consistently achieves the best BLEU and ASR-BLEU across French, German, and Spanish to English directions, outperforming strong recent S2ST baselines.

语音翻译多语言大模型模块化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。