提出动态上下文卷积与子网正则化,提升手语识别泛化能力
DESign: Dynamic Context-Aware Convolution and Efficient Subnet Regularization for Continuous Sign Language Recognition
- 用动态卷积捕捉帧间动作特征,按上下文细粒度调整权重
- 在三个主流数据集上达到最新最优效果,准确率显著提升
- 无需额外计算开销,可无缝接入现有手语识别模型
当前连续手语识别方法难以应对样本多样性。尽管动态卷积适合此任务,但主要关注空间建模,忽略时序动态与上下文依赖。为此,本文提出DESign框架,包含动态上下文感知卷积(DCAC)与子网正则化连接时序分类(SR-CTC)。DCAC动态捕捉构成手势的帧间运动线索,基于上下文信息细粒度自适应卷积权重,提升模型对多样化手语行为的泛化能力。此外,发现现有方法训练时仅依赖少数帧更新参数,导致CTC学习过拟合于单一路径。SR-CTC通过子网络监督引导模型探索多样对齐路径,有效防止过拟合;其共享分类器策略增强多尺度一致性。值得注意的是,SR-CTC无推理开销,可无缝集成至现有模型提升性能。大量消融实验与可视化验证了方法有效性。在PHOENIX14、PHOENIX14-T、CSL-Daily等主流数据集上的结果表明,DESign达到当前最优水平。
原文摘要 · Abstract (English)
Current continuous sign language recognition (CSLR) methods struggle with handling diverse samples. Although dynamic convolutions are ideal for this task, they mainly focus on spatial modeling and fail to capture the temporal dynamics and contextual dependencies. To address this, we propose DESign, a novel framework that incorporates Dynamic Context-Aware Convolution (DCAC) and Subnet Regularization Connectionist Temporal Classification (SR-CTC). DCAC dynamically captures the inter-frame motion cues that constitute signs and uniquely adapts convolutional weights in a fine-grained manner based on contextual information, enabling the model to better generalize across diverse signing behaviors and boost recognition accuracy. Furthermore, we observe that existing methods still rely on only a limited number of frames for parameter updates during training, indicating that CTC learning overfits to a dominant path. To address this, SR-CTC regularizes training by applying supervision to subnetworks, encouraging the model to explore diverse CTC alignment paths and effectively preventing overfitting. A classifier-sharing strategy in SR-CTC further strengthens multi-scale consistency. Notably, SR-CTC introduces no inference overhead and can be seamlessly integrated into existing CSLR models to boost performance. Extensive ablations and visualizations further validate the effectiveness of the proposed methods. Results on mainstream CSLR datasets (i.e., PHOENIX14, PHOENIX14-T, CSL-Daily) demonstrate that DESign achieves state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。