用轻量适配器让语音模型同时听声辨音，降噪效果更好且参数少
Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models
- 通过适配器和视觉融合模块，让纯音频模型接入视觉信息
- 在多数噪声场景下性能接近全量微调，参数量减少88.5%
- 可灵活添加新噪声适配器，无视觉时仍可用原模型
本文提出一种基于适配器的多智能体视听语音识别方法，以预训练的Whisper模型为基础。通过引入视听融合模块与LoRa适配器，将视觉信息注入仅支持音频的模型中。适配器方法仅需训练少量参数，保持主模型不变。针对不同噪声类别与噪声等级，我们为每种场景训练专属适配器集，通过提前分类噪声场景选择最优适配器集，实现对多种噪声条件的最优覆盖。相比全量微调的顶尖方法，本方案在多数测试噪声类别与等级下性能接近，但可训练参数最多减少88.5%。该方法可扩展新增噪声适配器，且当无视觉输入时，仍可直接使用原始强大的ASR模型。
原文摘要 · Abstract (English)
We present an approach to Audio-Visual Speech Recognition that builds on a pre-trained Whisper model. To infuse visual information into this audio-only model, we extend it with an AV fusion module and LoRa adapters, one of the most up-to-date adapter approaches. One advantage of adapter-based approaches, is that only a relatively small number of parameters are trained, while the basic model remains unchanged. Common AVSR approaches train single models to handle several noise categories and noise levels simultaneously. Taking advantage of the lightweight nature of adapter approaches, we train noise-scenario-specific adapter-sets, each covering individual noise-categories or a specific noise-level range. The most suitable adapter-set is selected by previously classifying the noise-scenario. This enables our models to achieve an optimum coverage across different noise-categories and noise-levels, while training only a minimum number of parameters. Compared to a full fine-tuning approach with SOTA performance our models achieve almost comparable results over the majority of the tested noise-categories and noise-levels, with up to 88.5% less trainable parameters. Our approach can be extended by further noise-specific adapter-sets to cover additional noise scenarios. It is also possible to utilize the underlying powerful ASR model when no visual information is available, as it remains unchanged.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。