让语音模型更懂语调，输入端改造效果优于传统输出端优化
Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
- 用统一编码器将语音转为带语调信息的文本标记
- 3B和7B模型在语调理解任务中表现最佳，模态差距最小
- 仅需约1000小时音频即可训练，效率高适合实际应用
语音大模型（SLM）通常基于文本大模型（TLM）预训练权重构建，但仍存在显著模态差距。以往研究主要从输出端入手，使语音生成更接近文本风格，但差距依然存在。本文认为关键瓶颈在于输入端。提出TextPro-SLM：结合WhisperPro（统一语音编码器，可同步生成文本标记与语调嵌入）与经微调的LLM骨干网络，在保持原始TLM语义能力的同时，学习副语言理解能力。实验表明，TextPro-SLM在3B和7B规模下均达到领先SLM中的最低模态差距，且在副语言理解任务上表现优异。仅需约1000小时的LLM训练音频即可实现这些提升，表明从输入端减少模态差距既有效又数据高效。
原文摘要 · Abstract (English)
Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。