通过分段压缩语音特征，实现语音与文本模态高效融合
SSR: Alignment-Aware Modality Connector for Speech Language Models
- 基于语音-文本对齐，分段压缩语音特征以匹配文本粒度
- 在StoryCloze上提升10分，在Speech-MMLU上提升20分
- 两阶段训练防止预训练文本能力遗忘，适合语音理解任务
将语音融入预训练语言模型（SpeechLM）常面临长语音编码效率低和预训练文本模态灾难性遗忘的问题。本文提出SSR-Connector（分段语音表征连接器），利用语音-文本对齐，对语音特征进行分段与压缩，使其匹配文本嵌入的粒度。此外，引入两阶段训练流程，包括知识蒸馏与微调阶段，有效缓解灾难性遗忘。SSR-Connector在语音-文本模态融合中表现更优，持续提升语音理解能力（如StoryCloze提升10分，Speech-MMLU提升20分），同时保持预训练文本能力。
原文摘要 · Abstract (English)
Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We propose SSR-Connector (Segmented Speech Representation Connector) for better modality fusion. Leveraging speech-text alignments, our approach segments and compresses speech features to match the granularity of text embeddings. Additionally, we introduce a two-stage training pipeline that includes the distillation and fine-tuning phases to mitigate catastrophic forgetting. SSR-Connector outperforms existing mechanism for speech-text modality fusion, consistently achieving better speech understanding (e.g., +10 accuracy on StoryCloze and +20 on Speech-MMLU) while preserving pre-trained text ability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。