arXiv:2506.16073cs.CV2025-06中稿 · publication in Jou…被引 1

改进唇读模型的时序建模能力,减少信息丢失。

TD3Net: A temporal densely connected multi-dilated convolutional network for lipreading

  • 引入多膨胀卷积与密集跳跃连接,构建无盲区时序感受野。
  • 在LRW和LRW-1000数据集上达到顶尖精度,参数更少、计算量更低。
  • 适合需要高效高精度唇读系统的开发者使用。

词级唇读通常采用前后端分离的两阶段框架,其中后端常使用时序卷积网络(TCN)。尽管密集跳跃连接已用于缓解感受野密度不足问题,但其性能仍受限于唇部运动连续性导致的信息损失。为此,本文提出TD3Net,一种结合密集跳跃连接与多膨胀时序卷积的后端架构。通过为跳跃连接特征施加不同膨胀因子,实现无盲区的宽而密的感受野。在公开数据集LRW和LRW-1000上的实验表明,该方法在词级唇读任务中表现接近现有最先进水平,同时参数量和浮点运算次数显著降低。可视化结果也显示其能有效利用多样化的时序特征并保持时间连续性,在唇读系统中具有明显优势。代码已开源。

原文摘要 · Abstract (English)

The word-level lipreading approach typically employs a two-stage framework with separate frontend and backend architectures to model dynamic lip movements. Each component has been extensively studied, and in the backend architecture, temporal convolutional networks (TCNs) have been widely adopted in state-of-the-art methods. Recently, dense skip connections have been introduced in TCNs to mitigate the limited density of the receptive field, thereby improving the modeling of complex temporal representations. However, their performance remains constrained owing to potential information loss regarding the continuous nature of lip movements, caused by blind spots in the receptive field. To address this limitation, we propose TD3Net, a temporal densely connected multi-dilated convolutional network that combines dense skip connections and multi-dilated temporal convolutions as the backend architecture. TD3Net covers a wide and dense receptive field without blind spots by applying different dilation factors to skip-connected features. Experimental results on a word-level lipreading task using two large publicly available datasets, Lip Reading in the Wild (LRW) and LRW-1000, indicate that the proposed method achieves performance comparable to state-of-the-art methods. It achieved higher accuracy with fewer parameters and lower floating-point operations compared to existing TCN-based backend architectures. Moreover, visualization results suggest that our approach effectively utilizes diverse temporal features while preserving temporal continuity, presenting notable advantages in lipreading systems. The code is available at our GitHub repository (https://github.com/Leebh-kor/TD3Net).

唇读时序建模卷积网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。