用扩散模型生成手语,速度更快、动作更自然。
MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation
- 采用掩码扩散架构,支持并行生成和双向上下文建模。
- 在多个数据集上表现领先,生成速度提升40%。
- 适合需要高效手语合成的应用,如无障碍沟通系统。
手语生成(SLG)旨在将书面文本转化为富有表现力的手语动作,帮助聋哑人群体跨越交流障碍。现有方法多基于自回归语言模型,存在单向上下文建模与逐词生成慢的问题。为此,本文提出MaDiS,一种基于掩码扩散的语言模型,可捕捉双向依赖并实现高效的并行多词生成。我们设计了三级跨模态预训练方案,联合学习词级、潜在表示及3D物理空间三个层次的信号,充分利用多层级手语表征。为加速微调阶段收敛,提出一种带时间检查点的新型解掩码策略,以粗到精的方式重构生成过程,使解掩码顺序的组合复杂度降低超过10^41倍。此外,引入分部位混合嵌入层,通过可学习门控与优化码本有效融合不同部位手语标记的信息。在CSL-Daily、Phoenix-2014T和How2Sign数据集上的大量实验表明,MaDiS在多种指标(包括DTW误差及新提出的SiBLEU、SiCLIP)上均取得最优表现,同时吞吐量提高40%。代码与模型将公开发布。
原文摘要 · Abstract (English)
Sign language generation (SLG) aims to translate written texts into expressive sign motions, bridging communication barriers for the Deaf and Hard-of-Hearing communities. Recent studies formulate SLG within the language modeling framework using autoregressive language models, which suffer from unidirectional context modeling and slow token-by-token inference. To address these limitations, we present MaDiS, a masked-diffusion-based language model for SLG that captures bidirectional dependencies and supports efficient parallel multi-token generation. We further introduce a tri-level cross-modal pretraining scheme that jointly learns from token-, latent-, and 3D physical-space objectives to leverage complementary, multi-level sign representations. To accelerate model convergence in the fine-tuning stage, we design a novel unmasking strategy with temporal checkpoints, which restructures generation in a coarse-to-fine manner and reduces the combinatorial complexity of unmasking orders by over $10^{41}$ times. In addition, a mixture-of-parts embedding layer is developed to effectively fuse information stored in different part-wise sign tokens through a learnable gate and well-optimized codebooks. Extensive experiments on CSL-Daily, Phoenix-2014T, and How2Sign demonstrate that MaDiS achieves superior performance across multiple metrics, including DTW error and two newly introduced metrics, SiBLEU and SiCLIP, while delivering a 40\% higher throughput. Code and models will be publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。