用扩散模型生成手语视频间的自然过渡帧，让动作更连贯
Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observation
- 用随机掩码模拟缺失帧，将无监督问题转为有监督训练
- 在三个数据集上生成视频的连续性提升，过渡更自然流畅
- 适合做手语合成、无障碍传播与数字人应用的开发者
从离散手语片段生成连续视频面临过渡不自然的挑战。传统拼接方法常导致突兀切换，破坏视频连贯性。为此，我们提出 Sign-D2C 框架，采用条件扩散模型生成上下文一致的过渡帧，实现手语序列的无缝衔接。通过在长时手语视频中随机掩码片段，将无监督的过渡帧生成转化为有监督训练任务。模型通过去噪高斯噪声并以邻近手语观测为条件，学习预测被遮蔽帧，可处理复杂非结构化过渡。推理时，采用线性插值填充初始缺失帧，为扩散模型迭代优化提供稳定基础。在 PHOENIX14T、USTC-CSL100 与 USTC-SLR500 数据集上的大量实验表明，该方法能有效生成连贯、自然的手语视频。
原文摘要 · Abstract (English)
Generating continuous sign language videos from discrete segments is challenging due to the need for smooth transitions that preserve natural flow and meaning. Traditional approaches that simply concatenate isolated signs often result in abrupt transitions, disrupting video coherence. To address this, we propose a novel framework, Sign-D2C, that employs a conditional diffusion model to synthesize contextually smooth transition frames, enabling the seamless construction of continuous sign language sequences. Our approach transforms the unsupervised problem of transition frame generation into a supervised training task by simulating the absence of transition frames through random masking of segments in long-duration sign videos. The model learns to predict these masked frames by denoising Gaussian noise, conditioned on the surrounding sign observations, allowing it to handle complex, unstructured transitions. During inference, we apply a linearly interpolating padding strategy that initializes missing frames through interpolation between boundary frames, providing a stable foundation for iterative refinement by the diffusion model. Extensive experiments on the PHOENIX14T, USTC-CSL100, and USTC-SLR500 datasets demonstrate the effectiveness of our method in producing continuous, natural sign language videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。