用符号当令牌,让AI自动生成多语言手语动画。
Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator
- 将手语拆解为身体部位的离散符号,用预训练语言模型生成3D手语动效。
- 多头并行解码提升效率,检索增强使手语词义更精准。
- 适合做无障碍沟通系统或跨语言手语生成研究者使用。
手语是一种包含自然语言所有语言特征的视觉语言,是听障人群的主要交流方式。尽管已有研究成功将预训练语言模型(LM)应用于手语翻译(手语转文字),但反向任务——手语生成(文字转手语)仍鲜有探索。本文提出多语言手语模型SOKE,可基于文本输入自回归生成3D手语虚拟角色动作。为实现手语与语言模型对齐,我们采用解耦分词器,将连续手语动作分解为表示不同身体部位的符号序列。在解码阶段,不同于现有方法将各部位符号拼成单一序列逐个预测,我们提出多头并行解码机制,可同时预测多个符号,提升推理效率并保持跨身体部位的有效信息融合。为进一步简化生成过程,提出检索增强的手语生成方法,引入外部手语词典作为辅助条件,显著提高生成手语的准确性。大量定性和定量评估验证了SOKE的有效性。
原文摘要 · Abstract (English)
Sign language is a visual language that encompasses all linguistic features of natural languages and serves as the primary communication method for the deaf and hard-of-hearing communities. Although many studies have successfully adapted pretrained language models (LMs) for sign language translation (sign-to-text), the reverse task-sign language generation (text-to-sign)-remains largely unexplored. In this work, we introduce a multilingual sign language model, Signs as Tokens (SOKE), which can generate 3D sign avatars autoregressively from text inputs using a pretrained LM. To align sign language with the LM, we leverage a decoupled tokenizer that discretizes continuous signs into token sequences representing various body parts. During decoding, unlike existing approaches that flatten all part-wise tokens into a single sequence and predict one token at a time, we propose a multi-head decoding method capable of predicting multiple tokens simultaneously. This approach improves inference efficiency while maintaining effective information fusion across different body parts. To further ease the generation process, we propose a retrieval-enhanced SLG approach, which incorporates external sign dictionaries to provide accurate word-level signs as auxiliary conditions, significantly improving the precision of generated signs. Extensive qualitative and quantitative evaluations demonstrate the effectiveness of SOKE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。