用CTC引导的教师强制法,让语音识别训练更快更稳。
Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition
- 用CTC伪标签直接驱动解码器生成注意力目标,一步完成。
- 训练时间减半,在长序列和噪声下表现更稳健,准确率领先。
- 适合追求高效、鲁棒的统一语音识别系统的研究者。
统一语音识别(USR)是一种半监督框架,旨在用单一模型处理音频、视觉及音视频语音识别任务,在分布内基准上达到顶尖性能。然而,其依赖自回归伪标签导致训练成本高,且CTC与注意力分支解耦监督易引发自我强化错误,尤其在长序列、噪声或未见领域下表现不佳。本文提出基于CTC的教师强制方法:将贪婪解码的CTC伪标签输入解码器,单次前向传播生成注意力目标。尽管全局可能不连贯,但在伪标签设定下能实现高效知识迁移。由于CTC与注意力伪标签长度一致,解码器可同时预测二者,兼顾CTC的鲁棒性与注意力的表达力,无需昂贵束搜索。进一步提出混合采样策略缓解解码器仅依赖CTC输入的暴露偏差。所提方法名为USR 2.0,训练时间减半,对分布外输入更具鲁棒性,在LRS3、LRS2和WildVSR上超越原有USR及模态特定自监督基线,达到最新水平。
原文摘要 · Abstract (English)
Unified Speech Recognition (USR) has emerged as a semi-supervised framework for training a single model for audio, visual, and audiovisual speech recognition, achieving state-of-the-art results on in-distribution benchmarks. However, its reliance on autoregressive pseudo-labelling makes training expensive, while its decoupled supervision of CTC and attention branches increases susceptibility to self-reinforcing errors, particularly under distribution shifts involving longer sequences, noise, or unseen domains. We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder to generate attention targets in a single forward pass. Although these can be globally incoherent, in the pseudo-labelling setting they enable efficient and effective knowledge transfer. Because CTC and CTC-driven attention pseudo-labels have the same length, the decoder can predict both simultaneously, benefiting from the robustness of CTC and the expressiveness of attention without costly beam search. We further propose mixed sampling to mitigate the exposure bias of the decoder relying solely on CTC inputs. The resulting method, USR 2.0, halves training time, improves robustness to out-of-distribution inputs, and achieves state-of-the-art results on LRS3, LRS2, and WildVSR, surpassing USR and modality-specific self-supervised baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。