仅用一块GPU即可生成手语视频，低成本实现文本到手语的快速转换。
Text2Sign: A Single-GPU Diffusion Baseline for Text-to-Sign Language Video Generation
- 采用冻结视觉语言编码器与3D时空分离注意力，降低计算开销。
- 在64×64分辨率下，32帧视频生成耗时12.6秒，峰值显存仅3.12GB。
- 适合研究者快速验证手语生成想法，但仅限短时低分辨率输出。
手语是数百万听障人士的主要交流方式，但文本转手语视频生成成本高昂，因视频扩散模型训练和评估代价大。本文提出Text2Sign，一种基于单块NVIDIA L4 GPU的短时手语视频生成文本条件扩散模型。该模型结合冻结的视觉-语言文本编码器与3D编码器-解码器，并采用因子化时空注意力，在保持动作连贯性的同时降低全视频注意力成本。我们对比了卷积与Transformer骨干网络、冻结预训练与任务特定文本编码器、因子化与完整注意力。在签员独立的How2Sign数据集上，最佳短周期消融实验验证损失为0.0648，长周期检查点达0.00999。在紧凑评估片段中，该模型使用8步DDIM采样（引导尺度5.0）获得SSIM 0.2403±0.0238，PSNR 15.11±0.42 dB，时间一致性1.0000±0.0000。每32帧64×64视频生成耗时12.60秒（2.54帧/秒），峰值推理显存3.12GB。持留去噪审计显示提示敏感度弱：移除文本使晚期损失从0.9875升至0.9891，打乱提示表现与正确提示相近。冻结文本条件虽提升短预算验证损失，但提示区分能力仍有限。系统仅支持低分辨率短片段生成，缺乏专家语言评估，应视为单GPU研究基线而非完整手语生产系统。代码见https://github.com/xiaruize0911/text2sign。
原文摘要 · Abstract (English)
Sign language is a primary communication channel for millions of Deaf and hard-of-hearing people, yet text-to-signer video generation remains costly because video diffusion models are expensive to train and evaluate. This paper presents Text2Sign, a text-conditioned diffusion model for short sign-language clips that runs on a single NVIDIA L4 GPU. It combines a frozen vision-language text encoder with a 3D encoder-decoder and factorized spatiotemporal attention to reduce the cost of full-video attention while preserving motion coherence. We compare convolution-only and transformer-style backbones, frozen pretrained and task-specific text encoders, and factorized versus full attention. On a signer-disjoint How2Sign split, the best short-run ablation reaches a validation loss of 0.0648, while a longer-run checkpoint reaches 0.00999. On a compact evaluation slice, the latter achieves an SSIM of $0.2403 \pm 0.0238$, a PSNR of $15.11 \pm 0.42$ dB, and temporal consistency of $1.0000 \pm 0.0000$ using 8-step DDIM sampling with a guidance scale of 5.0. It generates a 32-frame, $64 \times 64$ clip in 12.60 seconds, or 2.54 frames per second, with peak inference memory of 3.12 GB. A held-out denoising audit shows only weak prompt sensitivity: removing text increases late-timestep loss from 0.9875 to 0.9891, while shuffled prompts perform similarly to correct prompts. Frozen text conditioning therefore improves short-budget validation loss, but prompt-specific separation remains limited. The system is restricted to low-resolution, short clips and lacks expert linguistic evaluation, so it should be viewed as a single-GPU research baseline rather than a complete sign-language production system. Code is available at https://github.com/xiaruize0911/text2sign.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。