轻量级视觉语音识别模型,低资源下仍保持高准确率。
Designing Practical Models for Isolated Word Visual Speech Recognition
- 采用轻量化模块设计,优化时序卷积网络结构
- 在最大英文词汇公开数据集上达到高识别准确率
- 适合资源受限场景,如医疗辅助与嵌入式设备
视觉语音识别(VSR)系统仅通过视频序列解码口语词。此类系统在音频受损或不可用时,常作为辅助手段用于医疗协助和人机交互。现有系统依赖深度神经网络提取有效特征,虽性能优异,但计算开销大,硬件需求高,限制了实际部署。本文提出轻量级端到端VSR架构,基于图像分类中高效模型的基准测试,结合轻量化模块设计于时序卷积网络主干。构建多个低资源消耗但性能强劲的统一模型。在最大公开英语词汇数据集上验证了其有效性与实用性。代码与训练模型将公开共享。
原文摘要 · Abstract (English)
Visual speech recognition (VSR) systems decode spoken words from an input sequence using only the video data. Practical applications of such systems include medical assistance as well as human-machine interactions. A VSR system is typically employed in a complementary role in cases where the audio is corrupt or not available. In order to accurately predict the spoken words, these architectures often rely on deep neural networks in order to extract meaningful representations from the input sequence. While deep architectures achieve impressive recognition performance, relying on such models incurs significant computation costs which translates into increased resource demands in terms of hardware requirements and results in limited applicability in real-world scenarios where resources might be constrained. This factor prevents wider adoption and deployment of speech recognition systems in more practical applications. In this work, we aim to alleviate this issue by developing architectures for VSR that have low hardware costs. Following the standard two-network design paradigm, where one network handles visual feature extraction and another one utilizes the extracted features to classify the entire sequence, we develop lightweight end-to-end architectures by first benchmarking efficient models from the image classification literature, and then adopting lightweight block designs in a temporal convolution network backbone. We create several unified models with low resource requirements but strong recognition performance. Experiments on the largest public database for English words demonstrate the effectiveness and practicality of our developed models. Code and trained models will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。