仅用文本训练语音大模型，实现媲美多模态数据的效果
TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment
- 通过统一编码器将文本与语音映射到同一语义空间
- 纯文本训练下在多个语音任务上表现接近多模态基线
- 适合资源有限但需语音能力的场景
近期语音增强型语言模型在智能语音助手领域取得进展，但多数方法依赖大规模配对语音-文本数据及大量计算资源，存在可扩展性与可及性挑战。本文提出TESU-LLM框架，仅使用文本数据即可训练具备语音能力的语言模型。核心思路是利用统一编码器将语义等价的文本与语音输入映射至共享隐空间，并通过轻量级投影网络使编码器输出对齐语言模型的嵌入空间，从而实现从纯文本监督到语音推理的泛化。尽管仅基于文本训练，TESU-LLM在多个语音相关基准测试中表现优异，性能可比肩使用大规模多模态数据和大量算力训练的基线方法。结果表明该方法高效且有效，为无需语音数据构建语音大模型提供了可扩展路径。
原文摘要 · Abstract (English)
Recent advances in speech-enabled language models have shown promising results in building intelligent voice assistants. However, most existing approaches rely on large-scale paired speech-text data and extensive computational resources, which pose challenges in terms of scalability and accessibility. In this paper, we present \textbf{TESU-LLM}, a novel framework that enables training speech-capable language models using only text data. Our key insight is to leverage a unified encoder that maps semantically equivalent text and speech inputs to a shared latent space. By aligning the encoder output with the embedding space of a LLM via a lightweight projection network, we enable the model to generalize from text-only supervision to speech-based inference. Despite being trained exclusively on text, TESU-LLM achieves strong performance on various speech-related benchmarks, comparable to baseline methods trained with large-scale multimodal datasets and substantial computational resources. These results highlight the effectiveness and efficiency of our approach, offering a scalable path toward building speech LLMs without speech data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。