融合语言模型与扩散模型,用分步生成提升视频质量与语义一致性。
The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation
- 先用语义分词器将3D视觉特征压缩14000倍,生成离散语义令牌。
- 语言模型生成带高层语义关系的令牌,扩散模型逐帧细化成高清视频。
- 50亿参数模型在多个评测中超越130亿参数的Hunyuan Video和Sora等商用模型。
文本到视频生成近期发展依赖于两种竞争范式:自回归语言模型与扩散模型。然而两者各有局限:语言模型难以保证视觉质量且存在误差累积,扩散模型缺乏语义理解与因果建模能力。本文提出LanDiff,一种通过粗到精生成策略融合两者的混合框架。其架构包含三大创新:(1) 语义分词器将3D视觉特征压缩为紧凑的一维离散表示,实现约14,000倍压缩比;(2) 语言模型生成具有高层语义关系的语义令牌;(3) 流式扩散模型将粗粒度语义重构为高保真视频。实验表明,50亿参数的LanDiff在VBench T2V基准上取得85.43分,优于开源模型Hunyuan Video(13B)及商用模型Sora、Kling、Hailuo。此外,在长视频生成任务中也达到开源模型最优水平。演示可访问 https://landiff.github.io/。
原文摘要 · Abstract (English)
Recent advancements in text-to-video (T2V) generation have been driven by two competing paradigms: autoregressive language models and diffusion models. However, each paradigm has intrinsic limitations: language models struggle with visual quality and error accumulation, while diffusion models lack semantic understanding and causal modeling. In this work, we propose LanDiff, a hybrid framework that synergizes the strengths of both paradigms through coarse-to-fine generation. Our architecture introduces three key innovations: (1) a semantic tokenizer that compresses 3D visual features into compact 1D discrete representations through efficient semantic compression, achieving a $\sim$14,000$\times$ compression ratio; (2) a language model that generates semantic tokens with high-level semantic relationships; (3) a streaming diffusion model that refines coarse semantics into high-fidelity videos. Experiments show that LanDiff, a 5B model, achieves a score of 85.43 on the VBench T2V benchmark, surpassing the state-of-the-art open-source models Hunyuan Video (13B) and other commercial models such as Sora, Kling, and Hailuo. Furthermore, our model also achieves state-of-the-art performance in long video generation, surpassing other open-source models in this field. Our demo can be viewed at https://landiff.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。