无需姿态表示,直接文本生成手语视频,速度提升3倍。
FlashSign: Pose-Free Guidance for Efficient Sign Language Video Generation
- 用扩散模型直接从文本生成手语视频,跳过中间姿态步骤。
- 引入可训练滑动块注意力机制,推理速度提升3.07倍。
- 适合实时手语合成系统开发,尤其关注无障碍沟通应用。
手语在聋哑人群体与听力正常者之间沟通中至关重要。现有手语视频生成模型常依赖复杂的中间姿态表示,限制了灵活性和效率。本文提出一种新的无姿态框架,实现手语视频的实时生成。方法通过基于先进扩散模型的无姿态生成器,直接学习文本到手势的隐式对齐关系,无需姿态估计;同时引入可训练滑动块注意力(T-STA)机制,利用时空局部性加速推理。与以往训练自由稀疏方法不同,T-STA将可训练稀疏性融入训练与推理全过程,确保一致性并消除训练-测试差异。该方法显著降低计算开销,同时保持高质量生成,使实时部署成为可能。实验表明,视频生成速度提升3.07倍,且不损失质量。本工作为高质、实时、无姿态的手语合成开辟新路径,潜在应用于多元社区的包容性通信工具。代码已开源:https://github.com/AIGeeksGroup/FlashSign。
原文摘要 · Abstract (English)
Sign language plays a crucial role in bridging communication gaps between the deaf and hard-of-hearing communities. However, existing sign language video generation models often rely on complex intermediate representations, which limits their flexibility and efficiency. In this work, we propose a novel pose-free framework for real-time sign language video generation. Our method eliminates the need for intermediate pose representations by directly mapping natural language text to sign language videos using a diffusion-based approach. We introduce two key innovations: (1) a pose-free generative model based on the a state-of-the-art diffusion backbone, which learns implicit text-to-gesture alignments without pose estimation, and (2) a Trainable Sliding Tile Attention (T-STA) mechanism that accelerates inference by exploiting spatio-temporal locality patterns. Unlike previous training-free sparsity approaches, T-STA integrates trainable sparsity into both training and inference, ensuring consistency and eliminating the train-test gap. This approach significantly reduces computational overhead while maintaining high generation quality, making real-time deployment feasible. Our method increases video generation speed by 3.07x without compromising video quality. Our contributions open new avenues for real-time, high-quality, pose-free sign language synthesis, with potential applications in inclusive communication tools for diverse communities. Code: https://github.com/AIGeeksGroup/FlashSign.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。