arXiv:2503.15914cs.CV2025-03被引 7

用文本生成手语动作序列,精度高且语义一致。

Text-Driven Diffusion Model for Sign Language Production

  • 基于扩散模型,文本条件控制手语动作生成
  • 联合位置与骨骼方向损失,提升动作准确性
  • 适合手语合成与无障碍交互研究者

我们提出Text-driven Diffusion Model(TDM)框架,用于从文本生成语义对齐的手语动作序列。训练阶段,通过编码器将文本输入融入扩散模型,生成手语姿态序列;采用联合位置损失函数 $L_{joint}$ 和骨骼方向损失函数 $L_{bone}$,分别最小化生成姿态与真实姿态的关节位置差异及骨骼朝向偏差。推理阶段,模型从噪声序列出发,在文本约束下逐步优化生成语义一致的动作序列。该方法在挑战赛中取得20.17的BLEU-1得分,位列第二。

原文摘要 · Abstract (English)

We introduce the hfut-lmc team's solution to the SLRTP Sign Production Challenge. The challenge aims to generate semantically aligned sign language pose sequences from text inputs. To this end, we propose a Text-driven Diffusion Model (TDM) framework. During the training phase, TDM utilizes an encoder to encode text sequences and incorporates them into the diffusion model as conditional input to generate sign pose sequences. To guarantee the high quality and accuracy of the generated pose sequences, we utilize two key loss functions. The joint loss function L_{joint} is used to precisely measure and minimize the differences between the joint positions of the generated pose sequences and those of the ground truth. Similarly, the bone orientation loss function L_{bone} is instrumental in ensuring that the orientation of the bones in the generated poses aligns with the actual, correct orientations. In the inference stage, the TDM framework takes on a different yet equally important task. It starts with noisy sequences and, under the strict constraints of the text conditions, gradually refines and generates semantically consistent sign language pose sequences. Our carefully designed framework performs well on the sign language production task, and our solution achieves a BLEU-1 score of 20.17, placing second in the challenge.

手语生成扩散模型动作合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。