用LLM提升视频生成对文本的理解能力，让短句描述也能精准出片。
Mimir: Improving Video Diffusion Models for Precise Text Understanding
- 设计新型融合模块，协调文本编码器与LLM输出特征
- 在短文本和动态变化场景下显著提升文本匹配度
- 适合需要精准控制视频内容的创作者或研究者
文本是视频生成中的关键控制信号，因其叙事特性而至关重要。现有视频扩散模型虽借用文本编码器特征，但文本理解能力有限。近期大语言模型（LLMs）的成功表明，仅解码器架构的Transformer在文本到视频（T2V）生成中具有三大优势：更强的可扩展性带来更精确的文本理解、通过预测下一个词实现超越输入文本的想象能力、以及通过指令微调灵活响应用户偏好。然而，两种不同文本建模范式间的特征分布差异阻碍了LLM在现有T2V模型中的直接应用。本文提出Mimir，一种端到端训练框架，包含经过精心设计的令牌融合模块，以调和文本编码器与LLM的输出。该设计使T2V模型既能利用已学习的视频先验知识，又能充分发挥LLM的文本处理能力。大量定量与定性结果表明，Mimir在生成高质量视频并实现优秀文本理解方面效果显著，尤其在处理短文本描述和运动变化场景时表现突出。项目主页：https://lucaria-academy.github.io/Mimir/
原文摘要 · Abstract (English)
Text serves as the key control signal in video generation due to its narrative nature. To render text descriptions into video clips, current video diffusion models borrow features from text encoders yet struggle with limited text comprehension. The recent success of large language models (LLMs) showcases the power of decoder-only transformers, which offers three clear benefits for text-to-video (T2V) generation, namely, precise text understanding resulting from the superior scalability, imagination beyond the input text enabled by next token prediction, and flexibility to prioritize user interests through instruction tuning. Nevertheless, the feature distribution gap emerging from the two different text modeling paradigms hinders the direct use of LLMs in established T2V models. This work addresses this challenge with Mimir, an end-to-end training framework featuring a carefully tailored token fuser to harmonize the outputs from text encoders and LLMs. Such a design allows the T2V model to fully leverage learned video priors while capitalizing on the text-related capability of LLMs. Extensive quantitative and qualitative results demonstrate the effectiveness of Mimir in generating high-quality videos with excellent text comprehension, especially when processing short captions and managing shifting motions. Project page: https://lucaria-academy.github.io/Mimir/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。