用极少音视频标记实现高效语音识别，适合资源受限场景。
MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens
- 早融合+动态令牌分配，按说话速度智能压缩输入。
- 每秒仅3.5个标记，词错误率低至0.72%（LRS3数据集）。
- 相比前人方案减少86%标记量，计算量降低35.7%。
音频-视觉语音识别（AVSR）通过融合听觉与视觉信息，在嘈杂环境下实现鲁棒语音识别。然而，现有基于大语言模型（LLM）的AVSR系统因处理高时序分辨率的音视频信号而带来高昂计算开销。本文提出一种高效多模态语音LLM框架，以最小化标记长度同时保留关键语言内容。方法包括：早期音视频融合模块实现特征高效整合，音视频语音Q-Former根据输入时长动态分配标记，以及结合语音速率预测器的优化查询分配策略，按实际语速调整标记数量。在LRS3数据集上的大量实验表明，该方法达到当前最优性能，词错误率（WER）为0.72%，且每秒仅使用3.5个标记。相比此前多模态语音LLM框架，标记量减少86%，计算量（FLOPs）降低35.7%。
原文摘要 · Abstract (English)
Audio-Visual Speech Recognition (AVSR) achieves robust speech recognition in noisy environments by combining auditory and visual information. However, recent Large Language Model (LLM) based AVSR systems incur high computational costs due to the high temporal resolution of audio-visual speech processed by LLMs. In this work, we introduce an efficient multimodal speech LLM framework that minimizes token length while preserving essential linguistic content. Our approach employs an early AV-fusion module for streamlined feature integration, an audio-visual speech Q-Former that dynamically allocates tokens based on input duration, and a refined query allocation strategy with a speech rate predictor to adjust token allocation according to speaking speed of each audio sample. Extensive experiments on the LRS3 dataset show that our method achieves state-of-the-art performance with a WER of 0.72% while using only 3.5 tokens per second. Moreover, our approach not only reduces token usage by 86% compared to the previous multimodal speech LLM framework, but also improves computational efficiency by reducing FLOPs by 35.7%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。