一个统一模型同时处理听觉、视觉和音视频语音识别,效率更高。
Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models
- 用多粒度训练+参数高效适配,单模型支持多种语音识别任务。
- 在LRS2/LRS3上性能媲美甚至超过现有方法,训练资源减少超50%。
- 适合需要低资源部署的多模态语音识别场景,如智能设备或噪声环境。
大型语言模型(LLMs)在听觉语音识别(ASR)、视觉语音识别(VSR)和音视频语音识别(AVSR)等多模态任务中取得了显著进展。然而,现有基于LLM的方法通常独立训练不同任务的模型,导致计算与部署资源消耗大,并忽视跨任务协同潜力。同时,这些方法依赖固定速率的词元压缩,难以灵活权衡精度与效率。为此,本文提出Omni-AVSR,一种统一的音视频大语言模型,结合高效的多粒度训练与参数高效适配。具体地,将套娃式表征学习范式引入多粒度音视频训练,降低训练资源开销;并探索三种基于LoRA的骨干模型适配策略,平衡共享与任务特异性。在LRS2和LRS3数据集上的实验表明,Omni-AVSR在保持相当或更优精度的同时,仅需极低的训练与部署资源。模型在声学噪声下仍具鲁棒性,并分析了模型规模增长时的扩展行为,揭示性能与效率间的权衡关系。
原文摘要 · Abstract (English)
Large language models (LLMs) have recently achieved impressive results in speech recognition across multiple modalities, including Auditory Speech Recognition (ASR), Visual Speech Recognition (VSR), and Audio-Visual Speech Recognition (AVSR). Despite this progress, current LLM-based approaches typically address each task independently, training separate models that raise computational and deployment resource use while missing potential cross-task synergies. They also rely on fixed-rate token compression, which restricts flexibility in balancing accuracy with efficiency. These limitations highlight the need for a unified framework that can support ASR, VSR, and AVSR while enabling elastic inference. To this end, we present Omni-AVSR, a unified audio-visual LLM that combines efficient multi-granularity training with parameter-efficient adaptation. Specifically, we adapt the matryoshka representation learning paradigm to efficiently train across multiple audio and visual granularities, reducing its inherent training resource use. Furthermore, we explore three LoRA-based strategies for adapting the backbone LLM, balancing shared and task-specific specialization. Experiments on LRS2 and LRS3 show that Omni-AVSR achieves comparable or superior accuracy to state-of-the-art baselines while training a single model at substantially lower training and deployment resource use. The model also remains robust under acoustic noise, and we analyze its scaling behavior as LLM size increases, providing insights into the trade-off between performance and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。