将语音理解能力蒸馏到LLM中,实现高效端到端语音对话。
AuRA: Internalizing Audio Understanding into LLMs as LoRA

- 用轻量级嵌入层将语音同时输入教师ASR和学生LLM,逐层蒸馏对齐
- 在多个评测上优于串联系统与大型多模态模型,推理更快更准
- 适合需要低延迟语音交互的场景,如智能助手、实时翻译
为克服现有语音-语言模型依赖串行转录或昂贵多模态训练的问题,本文提出AuRA,通过知识蒸馏将语音编码能力内化至大语言模型。该方法将相同语音输入至预训练ASR编码器(教师)和LoRA微调的LLM(学生),通过逐层蒸馏使学生隐藏状态对齐教师表示,从而在不重新训练语音模型的前提下,实现轻量级语音理解能力注入。相比串行管道与桥接方法,AuRA支持更紧密的语音-语言联合建模与并行推理,同时复用已有预训练模型,避免大规模多模态训练。在多个语音-语言基准测试中,AuRA持续优于串行系统、基线适配方法以及大型多模态模型,在效果与效率上均表现更优。
原文摘要 · Abstract (English)
Recent efforts to extend large language models (LLMs) to speech inputs typically rely on cascaded ASR-LLM pipelines, end-to-end speech-language models, or bridge/distillation-based adaptation. While these routes respectively reuse strong pretrained components, enable native speech-language interaction, or offer lightweight adaptation, they often suffer from transcript-interface latency, costly multimodal training, or sequential speech-language coupling. To address these limitations, we present AuRA, a method that distills audio encoding capability into the LLM. Specifically, AuRA feeds the same speech input to an ASR encoder (as a teacher) and a LoRA-adapted LLM (as a student) through a lightweight audio embedding layer, and uses layer-wise distillation to align the student's hidden states with corresponding teacher representations, thereby internalizing speech representations into lightweight LLM-side adaptations. Compared with cascaded and serial bridge methods, AuRA enables tighter speech-language joint modeling and efficient parallel end-to-end inference, while also reusing pretrained speech and language models rather than requiring large-scale multimodal training. On multiple speech-language benchmarks, AuRA consistently outperforms cascaded systems, speech-to-LLM adaptation baselines, and large-scale speech-language and multimodal models in both effectiveness and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。