用固定数量隐变量压缩对话音频上下文，提升大模型语音识别效果
Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR
- 用可学习的隐变量替代历史对话的音频序列
- 在保持转录文本的同时，显著减少音频存储开销
- 适合长对话场景下的高效语音识别系统
基于大语言模型的语音识别系统通常孤立处理每个话语，难以利用对话上下文。本文研究多轮对话中跨模态上下文对语音识别的影响及其高效表示方式。实验发现，在多轮监督训练后，对话上下文主要提升对上下文相关实体的识别能力。然而，直接使用原始音频上下文会因历史音频标记序列随对话长度迅速增长而造成计算负担。为此，我们提出抽象压缩（Abstract Compression）：将历史对话的音频部分替换为固定数量的可学习隐变量，同时显式保留对应的转录文本。在域内和域外测试集上，该方法以更小的音频上下文存储开销，恢复了部分原始上下文条件带来的性能增益。我们还提供了对压缩设置及权衡的针对性分析。
原文摘要 · Abstract (English)
Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。