用多层嵌套结构实现音视频识别的动态计算适配
Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs
- 同一模型通过多粒度编码灵活分配音视频令牌
- 在不同算力下均达最优性能，超越固定压缩模型
- 适合资源受限场景下的实时音视频识别应用
音视频语音识别（AVSR）利用音频和视觉模态提升噪声环境下的鲁棒性。近年来大语言模型（LLMs）在语音识别领域表现出色，包括AVSR。然而，长语音表示导致LLM计算开销高。现有方法在输入前进行压缩，但高压缩率常损害精度。为此，我们提出Llama-MTSK，首个基于马特罗什卡（Matryoshka）的多模态LLM用于AVSR，可依据不同计算约束灵活调整音视频令牌分配。受马特罗什卡表示学习启发，模型以单一架构实现多粒度编码，无需多个独立模型。为实现高效微调，我们引入三种基于LoRA的策略，使用全局与尺度特定模块。在主要AVSR数据集上的评估表明，Llama-MTSK在性能上匹配或优于在固定压缩水平下训练的模型。
原文摘要 · Abstract (English)
Audio-Visual Speech Recognition (AVSR) leverages audio and visual modalities to improve robustness in noisy environments. Recent advances in Large Language Models (LLMs) show strong performance in speech recognition, including AVSR. However, the long speech representations lead to high computational costs for LLMs. Prior methods compress inputs before feeding them to LLMs, but high compression often harms accuracy. To address this, we propose Llama-MTSK, the first Matryoshka-based Multimodal LLM for AVSR, which flexibly adapts audio-visual token allocation under varying compute constraints. Inspired by Matryoshka Representation Learning, our model encodes representations at multiple granularities with a single architecture, avoiding the need for separate models. For efficient fine-tuning, we introduce three LoRA-based strategies using global and scale-specific modules. Evaluations on major AVSR datasets show Llama-MTSK matches or outperforms models trained at fixed compression levels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。