arXiv:2503.12953cs.CV2025-03中稿 · TMLR, 11/2025被引 2

通过帧级条件适配,提升文本到视频生成的连贯性。

Frame-wise Conditioning Adaptation for Fine-Tuning Diffusion Models in Text-to-Video Prediction

  • 设计帧级文本嵌入模块,动态生成每帧对应的文本条件。
  • 在多个数据集上实现新最优性能,显著改善视频时序连续性。
  • 适合关注视频生成连贯性的研究者与开发者使用。

文本到视频预测(TVP)是一项下游视频生成任务,要求模型根据初始视频帧和描述运动的文本生成后续帧。现有方法多聚焦于人类或机械臂操作物体的视频类别,通常基于预训练的文生图模型进行微调,导致生成视频缺乏时序连续性。借鉴更先进的文生视频(T2V)模型可进一步提升效果,但主流微调技术低秩适配(LoRA)表现不佳。本文提出帧级条件适配(FCA)策略,在模块中设计子模块,从输入文本生成逐帧文本嵌入,作为额外条件辅助生成,并将初始帧作为附加条件融入模型。我们系统评估了嵌入注入方式的有效性,通过大量消融实验验证设计选择,定性和定量分析均表明该方法在多个基准上达到新最佳性能。代码已开源。

原文摘要 · Abstract (English)

Text-video prediction (TVP) is a downstream video generation task that requires a model to produce subsequent video frames given a series of initial video frames and text describing the required motion. In practice TVP methods focus on a particular category of videos depicting manipulations of objects carried out by human beings or robot arms. Previous methods adapt models pre-trained on text-to-image tasks, and thus tend to generate video that lacks the required continuity. A natural progression would be to leverage more recent pre-trained text-to-video (T2V) models. This approach is rendered more challenging by the fact that the most common fine-tuning technique, low-rank adaptation (LoRA), yields undesirable results. In this work, we propose an adaptation-based strategy we label Frame-wise Conditioning Adaptation (FCA). Within the module, we devise a sub-module that produces frame-wise text embeddings from the input text, which acts as an additional text condition to aid generation. We use FCA to fine-tune the T2V model, which incorporates the initial frame(s) as an extra condition. We compare and discuss the more effective strategy for injecting such embeddings into the T2V model. We conduct extensive ablation studies on our design choices with quantitative and qualitative performance analysis. Our approach establishes a new state-of-the-art for the task of TVP. Our code is open-source at https://github.com/Cuberick-Orion/FCA .

文本到视频扩散模型视频生成微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。