让视频生成更懂文字描述，精准控制物体运动轨迹。
TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generation
- 通过文本视觉对齐，让模型理解文字中的物体及其运动
- 在多个数据集上达到当前最佳视频质量，减少物体消失问题
- 适合需要精确控制动作的视频生成场景
文本驱动图像到视频生成（TI2V）旨在根据首帧图像和对应文本描述生成可控视频。主要挑战在于：(i) 如何识别目标物体，并确保运动轨迹与文本描述一致；(ii) 如何提升生成视频的主观质量。为此，我们提出基于扩散模型的新框架TIV-Diffusion，通过对象中心的文本-视觉对齐，实现基于文本描述的物体运动精准控制与高质量视频生成。具体而言，利用尺度偏移调制融合文本与视觉知识，使模型感知文本描述的物体及其运动轨迹。此外，为缓解物体消失与错位问题，引入对象中心的文本-视觉对齐模块，解耦参考图像中的物体，并分别将文本特征与各物体对齐。基于上述创新，TIV-Diffusion在多个指标上优于现有TI2V方法，实现高质量视频生成。
原文摘要 · Abstract (English)
Text-driven Image to Video Generation (TI2V) aims to generate controllable video given the first frame and corresponding textual description. The primary challenges of this task lie in two parts: (i) how to identify the target objects and ensure the consistency between the movement trajectory and the textual description. (ii) how to improve the subjective quality of generated videos. To tackle the above challenges, we propose a new diffusion-based TI2V framework, termed TIV-Diffusion, via object-centric textual-visual alignment, intending to achieve precise control and high-quality video generation based on textual-described motion for different objects. Concretely, we enable our TIV-Diffuion model to perceive the textual-described objects and their motion trajectory by incorporating the fused textual and visual knowledge through scale-offset modulation. Moreover, to mitigate the problems of object disappearance and misaligned objects and motion, we introduce an object-centric textual-visual alignment module, which reduces the risk of misaligned objects/motion by decoupling the objects in the reference image and aligning textual features with each object individually. Based on the above innovations, our TIV-Diffusion achieves state-of-the-art high-quality video generation compared with existing TI2V methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。