一个模型搞定听觉、视觉和视听语音识别，性能更优且省资源。
Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
- 用统一训练策略让单个模型同时处理三种语音输入模态。
- 在LRS3、LRS2和WildVSR数据集上均达最新最佳性能。
- 提出贪心伪标签法，更好利用无标签数据，适合多模态语音研究者。
听觉、视觉和视听语音识别（ASR、VSR 和 AVSR）的研究传统上各自独立。即使近期一些自监督方法同时处理两种或全部三种任务，也通常生成独立模型，导致推理流程割裂、内存开销大且存在冗余。本文提出统一训练策略，证明单一模型联合训练可提升VSR与AVSR性能，克服从零开始训练的典型优化难题。此外，提出一种贪心伪标签方法，更有效地利用未标注样本，弥补现有自监督方法的不足。最后，在框架内构建自监督预训练方法，验证其与半监督方法的协同有效性。尽管使用单一模型完成所有任务，本方法在LRS3、LRS2及新发布的WildVSR数据集上，对ASR、VSR和AVSR均达到当前最优表现。代码与模型已开源。
原文摘要 · Abstract (English)
Research in auditory, visual, and audiovisual speech recognition (ASR, VSR, and AVSR, respectively) has traditionally been conducted independently. Even recent self-supervised studies addressing two or all three tasks simultaneously tend to yield separate models, leading to disjoint inference pipelines with increased memory requirements and redundancies. This paper proposes unified training strategies for these systems. We demonstrate that training a single model for all three tasks enhances VSR and AVSR performance, overcoming typical optimisation challenges when training from scratch. Moreover, we introduce a greedy pseudo-labelling approach to more effectively leverage unlabelled samples, addressing shortcomings in related self-supervised methods. Finally, we develop a self-supervised pre-training method within our framework, proving its effectiveness alongside our semi-supervised approach. Despite using a single model for all tasks, our unified approach achieves state-of-the-art performance compared to recent methods on LRS3 and LRS2 for ASR, VSR, and AVSR, as well as on the newly released WildVSR dataset. Code and models are available at https://github.com/ahaliassos/usr.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。