提出轻量协同框架,实现多语言语音翻译的低带宽与高隐私
Parameter- and Bandwidth-Efficient Edge--cloud Many-to-Many Speech-to-Text Translation

- 设备端保留轻量编码器,云端仅传输压缩张量
- 45种语言下性能达当前最佳,传输数据量减少超5倍
- 适合资源受限场景下的隐私敏感语音翻译应用
多模态大模型在语音到文本翻译(S2TT)中展现巨大潜力。然而现有部署方案面临严峻挑战:纯设备端模型受资源限制,集中式云系统则因传输原始语音数据导致带宽瓶颈与隐私风险。本文提出边缘-云协同的语音识别与翻译框架ESRT,采用多任务加权课程学习策略,训练出参数高效的ESRT-1B、ESRT-4B和ESRT-12B模型,有效缓解灾难性遗忘并提升多语言平衡。通过在设备端保留轻量语音编码器与适配器,仅向云端传输压缩张量,实现高效推理。在FLEURS数据集上的实验表明,ESRT在45种语言(45×44个方向)上达到当前最优性能。相较于原始音频,ESRT与ESRT-Lite分别将传输张量大小减少5.1倍和10.2倍,同时确保原始语音保留在本地,避免直接暴露于云端。代码与模型已开源,支持可复现的隐私友好型S2TT研究。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have demonstrated significant potential for speech-to-text translation (S2TT). However, existing deployment paradigms face critical challenges: pure on-device models suffer from resource constraints, while centralized cloud systems incur bandwidth bottlenecks and privacy risks by transmitting raw voice data. In this paper, we propose Edge--cloud Speech Recognition and Translation (ESRT), a parameter-efficient, bandwidth-efficient, and privacy-aware collaborative Edge--cloud MLLM framework. First, we introduce a multi-task weighted curriculum learning strategy to mitigate catastrophic forgetting, improve multilingual balance, and train parameter-efficient ESRT-1B, ESRT-4B, and ESRT-12B models. Second, we enable bandwidth-efficient Edge--cloud inference by retaining a lightweight speech encoder and adapter on the device and transmitting only a compressed tensor to the cloud. Extensive experiments on FLEURS demonstrate that ESRT models achieve state-of-the-art S2TT performance across 45 languages ($45 \times 44$ directions). Relative to raw audio, ESRT and ESRT-Lite reduce the transmitted tensor size by $5.1\times$ and $10.2\times$, respectively, while keeping raw speech on-device and avoiding its direct exposure to the cloud. The code and models are released to facilitate reproducible, privacy-aware S2TT research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。