用离散符号压缩多条件信息,生成更自然的手语视频。
Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization
- 将手部姿态和3D手势等细粒度信息转为离散符号,提升表达精度
- 在多个数据集上实现最高视频质量与语义一致性,超越现有方法
- 适合需要高保真手语生成的研究者或无障碍应用开发者
手语视频生成(SLVG)旨在从口语文本生成保持身份特征的手语视频。现有方法主要依赖单一粗粒度条件(如骨骼序列)作为翻译模型与视频生成模型之间的桥梁,限制了生成视频的自然度与表现力。为此,我们提出 SignViP 框架,通过引入多细粒度条件来提升生成保真度。SignViP 采用离散标记化范式,整合并表示细粒度条件(如细粒度姿态和3D手部动作)。该框架包含三个核心组件:(1) 与多条件编码器联合训练的符号视频扩散模型,学习封装细粒度运动与外观的连续嵌入;(2) 有限标量量化(FSQ)自编码器,进一步将这些嵌入压缩并量化为离散标记,实现紧凑条件表示;(3) 多条件标记翻译器,用于将口语文本翻译为离散多条件标记。推理时,标记翻译器先将文本转为离散标记,经 FSQ 自编码器解码为连续嵌入,再注入符号视频扩散模型以引导视频生成。实验表明,SignViP 在视频质量、时间连贯性和语义保真度等多项指标上均达到当前最优水平。
原文摘要 · Abstract (English)
Sign Language Video Generation (SLVG) seeks to generate identity-preserving sign language videos from spoken language texts. Existing methods primarily rely on the single coarse condition (\eg, skeleton sequences) as the intermediary to bridge the translation model and the video generation model, which limits both the naturalness and expressiveness of the generated videos. To overcome these limitations, we propose SignViP, a novel SLVG framework that incorporates multiple fine-grained conditions for improved generation fidelity. Rather than directly translating error-prone high-dimensional conditions, SignViP adopts a discrete tokenization paradigm to integrate and represent fine-grained conditions (\ie, fine-grained poses and 3D hands). SignViP contains three core components. (1) Sign Video Diffusion Model is jointly trained with a multi-condition encoder to learn continuous embeddings that encapsulate fine-grained motion and appearance. (2) Finite Scalar Quantization (FSQ) Autoencoder is further trained to compress and quantize these embeddings into discrete tokens for compact representation of the conditions. (3) Multi-Condition Token Translator is trained to translate spoken language text to discrete multi-condition tokens. During inference, Multi-Condition Token Translator first translates the spoken language text into discrete multi-condition tokens. These tokens are then decoded to continuous embeddings by FSQ Autoencoder, which are subsequently injected into Sign Video Diffusion Model to guide video generation. Experimental results show that SignViP achieves state-of-the-art performance across metrics, including video quality, temporal coherence, and semantic fidelity. The code is available at https://github.com/umnooob/signvip/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。