通过音素中间层提升唇语识别准确率,仅用极少标注数据达到顶尖水平。
VALLR: Visual ASR Language Model for Lip Reading
- 先预测音素序列,再用大语言模型生成完整句子
- 在LRS3上达到18.7%的词错误率,比之前方法少用99.4%标注数据
- 适合低资源场景下的高精度唇语识别任务
唇语识别(即视觉自动语音识别,V-ASR)是一项复杂任务,需仅凭视觉线索(如唇部运动和面部表情)解读语言。由于缺乏听觉信息,且不同音素可能呈现相同嘴型(共现音素模糊),该任务极具挑战性。现有方法通常直接从视觉信号预测词汇或字符,但受共音效应和音素模糊影响,错误率较高。本文提出一种两阶段、以音素为中心的V-ASR框架:首先使用带CTC头的视频变换器从视觉输入中预测紧凑的音素序列,降低任务复杂度并实现强说话人无关性;随后将音素序列输入微调的大语言模型,利用更广泛的语言上下文重构连贯的词语与句子。相比直接预测词汇的方法或依赖大规模多模态预训练的方案,本方法显式编码中间语言结构,同时具备高度数据效率。在两个挑战性数据集LRS2和LRS3上均实现领先性能,尤其在LRS3上达到18.7%的词错误率(WER),仅使用前一最佳方法0.6%的标注数据。
原文摘要 · Abstract (English)
Lip Reading, or Visual Automatic Speech Recognition (V-ASR), is a complex task requiring the interpretation of spoken language exclusively from visual cues, primarily lip movements and facial expressions. This task is especially challenging due to the absence of auditory information and the inherent ambiguity when visually distinguishing phonemes that have overlapping visemes where different phonemes appear identical on the lips. Current methods typically attempt to predict words or characters directly from these visual cues, but this approach frequently encounters high error rates due to coarticulation effects and viseme ambiguity. We propose a novel two-stage, phoneme-centric framework for Visual Automatic Speech Recognition (V-ASR) that addresses these longstanding challenges. First, our model predicts a compact sequence of phonemes from visual inputs using a Video Transformer with a CTC head, thereby reducing the task complexity and achieving robust speaker invariance. This phoneme output then serves as the input to a fine-tuned Large Language Model (LLM), which reconstructs coherent words and sentences by leveraging broader linguistic context. Unlike existing methods that either predict words directly-often faltering on visually similar phonemes-or rely on large-scale multimodal pre-training, our approach explicitly encodes intermediate linguistic structure while remaining highly data efficient. We demonstrate state-of-the-art performance on two challenging datasets, LRS2 and LRS3, where our method achieves significant reductions in Word Error Rate (WER) achieving a SOTA WER of 18.7 on LRS3 despite using 99.4% less labelled data than the next best approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。