用凸包约束让语音嵌入贴合语言模型,提升语音理解性能。
Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

- 将每帧语音表示为词嵌入的凸组合,强制对齐预训练语言模型空间。
- 在语音识别上相对降低48.7%错误率,情绪识别性能持平或超越单任务模型。
- 揭示语音信息依赖嵌入空间轨迹而非离散词元,适合研究冻结大模型多模态融合。
大型语言模型(LLM)为语音理解提供强大的推理基础,但将连续声学信号融入冻结的LLM仍具挑战。现有语音到LLM接口通常处于两个极端:要么强制近似离散词元对齐,虽利于转录却丢失副语言信息;要么学习无约束的连续表示,易偏离LLM输入空间,损害自回归解码。本文提出凸门(C-Gate),一种语音到LLM的桥梁,通过架构上的凸包约束,确保所有语音表示位于预训练LLM的输入嵌入流形内。具体而言,每帧语音被表示为词嵌入的凸组合,既保持与预训练LLM兼容性,又保留连续表达能力。在自动语音识别(ASR)和情感识别任务中,C-Gate实现强联合性能,使LibriSpeech的词错误率(WER)相对降低最高达48.7%,同时情感识别准确率与单任务模型相当或更优。分析表明:信息并非由离散词元身份携带,而是由嵌入空间中的时序轨迹承载。因果干预验证轨迹结构及对预训练嵌入流形的对齐均对性能至关重要。结果表明,几何结构而非词元离散性才是语音到LLM接口的核心设计因素,并为研究冻结LLM的多模态融合提供了可控范式。我们公开了检查点、逐样本输出、机制快照及干预工具集以供复现。
原文摘要 · Abstract (English)
Large language models (LLMs) provide a powerful reasoning backbone for speech understanding, but integrating continuous acoustic signals into a frozen LLM remains challenging. Existing speech-to-LLM interfaces typically operate at two extremes: either enforcing near-discrete token alignment, which benefits transcription but loses paralinguistic information, or learning unconstrained continuous representations, which can drift away from the LLM's input space and degrade autoregressive decoding. In this work, we propose Convex Gate (C-Gate), a speech-to-LLM bridge that constrains all speech representations to lie within the LLM's input embedding manifold with an architectural convex-hull constraint. Concretely, each frame is represented as a convex combination of token embeddings, ensuring compatibility with the pretrained LLM while preserving continuous expressivity. Across automatic speech recognition (ASR) and emotion recognition, C-Gate achieves strong joint performance, improving LibriSpeech WER by up to 48.7% relative while matching or exceeding single-task emotion accuracy. Beyond performance, our analysis reveals a key insight: information is not carried by discrete token identities, but by time-resolved trajectories in the embedding space. Causal interventions confirm that both the trajectory structure and alignment to the pretrained embedding manifold are critical for performance. These results suggest that geometry, rather than token discreteness, is the fundamental design factor in speech-to-LLM interfaces, and provide a controlled regime for studying multimodal integration in frozen LLMs. We release the checkpoint, per-sample outputs, mechanism dumps, and intervention suite for replication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。