用动态聚焦机制提升无声视频语音识别准确率
InfoSyncNet: Information Synchronization Temporal Convolutional Network for Visual Speech Recognition
- 设计非均匀量化模块，动态调整网络关注点
- 在LRW和LRW1000上分别达92.0%和60.7%准确率
- 适合做辅助技术与增强现实中的视觉语音识别
从无声视频中估计口语内容对辅助技术和增强现实应用至关重要。然而，由于序列间差异大且每段序列信息分布不均，将视频中的唇部运动序列映射到词语面临巨大挑战。为此，我们提出InfoSyncNet，一种结合定制数据增强的非均匀序列建模网络。其核心是在编码器与解码器之间引入非均匀量化模块，使网络能动态调整关注焦点，有效应对视觉语音数据的自然不一致性。此外，采用多种训练策略提升模型对光照变化和说话人姿态差异的鲁棒性。在LRW和LRW1000数据集上的全面实验表明，InfoSyncNet取得92.0%和60.7%的Top-1准确率，达到新最优水平。代码已公开。
原文摘要 · Abstract (English)
Estimating spoken content from silent videos is crucial for applications in Assistive Technology (AT) and Augmented Reality (AR). However, accurately mapping lip movement sequences in videos to words poses significant challenges due to variability across sequences and the uneven distribution of information within each sequence. To tackle this, we introduce InfoSyncNet, a non-uniform sequence modeling network enhanced by tailored data augmentation techniques. Central to InfoSyncNet is a non-uniform quantization module positioned between the encoder and decoder, enabling dynamic adjustment to the network's focus and effectively handling the natural inconsistencies in visual speech data. Additionally, multiple training strategies are incorporated to enhance the model's capability to handle variations in lighting and the speaker's orientation. Comprehensive experiments on the LRW and LRW1000 datasets confirm the superiority of InfoSyncNet, achieving new state-of-the-art accuracies of 92.0% and 60.7% Top-1 ACC. The code is available for download (see comments).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。