arXiv:2608.24674cs.CV2026-08

让图文音生成快20倍,还能保持同步与质量。

TurboT2VA: Fast Large-Scale Text-to-Video-Audio Generation via Score-Regularized Consistency Distillation

论文配图:TurboT2VA: Fast Large-Scale Text-to-Video-Audio Generation via Score-Regularized Consistency Distillation
图 1 · 摘自论文原文
  • 用分模态归一化和渐进式训练提升多模态生成稳定性。
  • 4步蒸馏后生成速度从50.52秒降至2.51秒,提速20.1倍。
  • 适合需要高速高质视频音频生成的开发者和研究者。

联合文本到视频音频生成可产出同步的视觉与听觉内容,但大模型的长采样轨迹和异构多模态计算使推理代价极高。本文提出TurboT2VA,一种用于190亿参数联合视频音频模型的蒸馏与推理框架。大规模T2VA蒸馏面临模态不平衡优化、大规模连续时间一致性训练困难及质量-多样性权衡问题。TurboT2VA通过分模态归一化和渐进式课程学习(离散一致性预热、连续一致性精炼、联合一致性-分布匹配)解决上述挑战。课程先建立稳定多样生成轨迹,再引入分布级优化。在LTX-2数据集上,四步蒸馏将生成延迟从50.52秒降至2.51秒(标准分辨率512×768),实现20.1倍加速,同时保持强视觉质量、音频保真度、多样性与音视频同步。进一步构建架构感知推理栈,结合受保护的W8A8量化、融合算子、压缩文本填充与模态感知稀疏注意力,保留密集跨模态与文本条件路径。在高分辨率部署设置(1024×1792)下,完整栈将生成延迟从318.74秒降至5.83秒(单块NVIDIA H20),实现54.67倍仅生成器加速。代码与演示见https://github.com/thu-ml/TurboDiffusion/tree/main/turbot2va。

原文摘要 · Abstract (English)

Joint text-to-video-audio generation produces synchronized visual and acoustic content, but the long sampling trajectories and heterogeneous multimodal computation of large models make inference prohibitively expensive. We present TurboT2VA, a distillation and inference framework for accelerating a 19B-parameter joint video-audio model. Large-scale T2VA distillation is challenged by modality-imbalanced optimization, the difficulty of continuous-time consistency training at scale, and the quality--diversity trade-off. TurboT2VA addresses these issues with per-modality normalization and a progressive curriculum comprising discrete consistency warm-up, continuous consistency refinement, and joint consistency--distribution matching. The curriculum first establishes a stable, diverse generation trajectory and only then introduces distribution-level refinement. On LTX-2, four-step distillation reduces generator latency from 50.52s to 2.51s at the standard evaluation resolution of 512$\times$768, achieving a 20.1$\times$ speedup while maintaining strong visual quality, audio fidelity, diversity, and video-audio synchronization. We further develop an architecture-aware inference stack that combines guarded W8A8 and fused operators, padded-text compaction, and modality-aware sparse attention while preserving dense cross-modal and text-conditioning paths. Under the high-resolution deployment setting at 1024$\times$1792, the complete stack reduces generator latency from 318.74s to 5.83s on one NVIDIA H20, achieving a 54.67$\times$ generator-only speedup. Inference code and generation demos are available at https://github.com/thu-ml/TurboDiffusion/tree/main/turbot2va.

视频生成多模态加速推理扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。