让语音模型真正听懂语音，而非仅把语音当文本处理
Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text
- 分离语音与文本长度差异，强化二者语义对齐
- 在多个评测中表现优于主流基线模型
- 适合研究语音-文本对齐与多模态模型的开发者
语音语言模型（SLMs）直接从语音生成文本，为端到端系统提供了替代方案。尽管近期取得进展，现有SLMs在指令遵循能力与跨任务泛化方面仍弱于文本语言模型。分析显示，当前SLMs中语音与文本表征虽下游性能良好，但结构对齐仍较薄弱，反映出连续、时变语音与离散文本间的结构性差异未被充分解决。为此，我们提出一个简单框架，将长度不匹配与语义对齐解耦，并促进语音与文本表征更紧密对应。在多个基准测试中的实验表明，该方法性能媲美强基线，凸显了在训练SLM时显式处理语音与文本结构性差异的重要性。代码已公开于https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text。
原文摘要 · Abstract (English)
Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。