让语音识别与大模型在手机等设备上高效协同,实现个性化语音交互。
Tiny-Align: Bridging Automatic Speech Recognition and Large Language Model on the Edge
- 提出轻量级跨模态对齐框架,实现语音与语言模型在边缘设备上的联合训练。
- 在8GB内存设备上实现50倍训练加速,对齐质量提升超50%。
- 适合需要本地化、个性化语音助手的移动和嵌入式场景。
将大型语言模型(LLM)与自动语音识别(ASR)结合部署在边缘设备(称为边缘ASR-LLM),可为用户提供强大的个性化语音交互能力。相比文本交互,边缘ASR-LLM支持更自然、无障碍的音频交互。然而,现有ASR-LLM模型主要在高性能计算环境训练,模型体积庞大,难以部署于边缘设备。更重要的是,为满足用户个性化需求,系统需能基于各用户独特的语音特征进行本地化训练。由于单独微调ASR或LLM会受限于模态特性导致效果不佳,端到端训练能实现音频特征与语言理解的无缝融合(跨模态对齐),从而在边缘设备上实现更高效、个性化的适配。然而,现有方法训练复杂且计算开销大,难以在资源受限设备上实现跨模态对齐。本文提出一种资源高效的跨模态对齐框架,首次在低资源边缘设备如NVIDIA Jetson Orin(8GB RAM)上实现高效ASR-LLM对齐,训练时间提速50倍,对齐质量提升超过50%。
原文摘要 · Abstract (English)
The combination of Large Language Models (LLM) and Automatic Speech Recognition (ASR), when deployed on edge devices (called edge ASR-LLM), can serve as a powerful personalized assistant to enable audio-based interaction for users. Compared to text-based interaction, edge ASR-LLM allows accessible and natural audio interactions. Unfortunately, existing ASR-LLM models are mainly trained in high-performance computing environments and produce substantial model weights, making them difficult to deploy on edge devices. More importantly, to better serve users' personalized needs, the ASR-LLM must be able to learn from each distinct user, given that audio input often contains highly personalized characteristics that necessitate personalized on-device training. Since individually fine-tuning the ASR or LLM often leads to suboptimal results due to modality-specific limitations, end-to-end training ensures seamless integration of audio features and language understanding (cross-modal alignment), ultimately enabling a more personalized and efficient adaptation on edge devices. However, due to the complex training requirements and substantial computational demands of existing approaches, cross-modal alignment between ASR audio and LLM can be challenging on edge devices. In this work, we propose a resource-efficient cross-modal alignment framework that bridges ASR and LLMs on edge devices to handle personalized audio input. Our framework enables efficient ASR-LLM alignment on resource-constrained devices like NVIDIA Jetson Orin (8GB RAM), achieving 50x training time speedup while improving the alignment quality by more than 50\%. To the best of our knowledge, this is the first work to study efficient ASR-LLM alignment on resource-constrained edge devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。