用时间缓冲帧实现视频扩散模型的高效可控生成。
Temporal In-Context Fine-Tuning with Temporal Reasoning for Versatile Control of Video Diffusion Models
- 将条件帧与目标帧拼接，插入渐进噪声的缓冲帧以匹配模型时间动态。
- 仅需10-30样本即可达到优秀效果，优于现有基线方法。
- 无需修改架构，适合小样本、多任务视频生成场景。
近期文本到视频扩散模型实现了高质量视频合成，但可控生成仍具挑战，尤其在数据和算力受限时。现有微调方法常依赖外部编码器或结构改动，需大量数据且仅支持空间对齐条件，灵活性与可扩展性差。本文提出时间上下文微调（TIC-FT），一种高效通用的预训练视频扩散模型适配方法。核心思想是沿时间轴拼接条件帧与目标帧，并插入噪声水平逐步升高的中间缓冲帧，实现平滑过渡，契合预训练模型的时间动态。TIC-FT无需架构变更，在仅10-30个样本下表现优异。我们在CogVideoX-5B与Wan-14B等大规模基础模型上验证了其在图像到视频、视频到视频生成等任务上的有效性。实验表明，TIC-FT在条件保真度与视觉质量上均超越现有基线，同时训练与推理效率高。更多结果见https://kinam0252.github.io/TIC-FT/
原文摘要 · Abstract (English)
Recent advances in text-to-video diffusion models have enabled high-quality video synthesis, but controllable generation remains challenging, particularly under limited data and compute. Existing fine-tuning methods for conditional generation often rely on external encoders or architectural modifications, which demand large datasets and are typically restricted to spatially aligned conditioning, limiting flexibility and scalability. In this work, we introduce Temporal In-Context Fine-Tuning (TIC-FT), an efficient and versatile approach for adapting pretrained video diffusion models to diverse conditional generation tasks. Our key idea is to concatenate condition and target frames along the temporal axis and insert intermediate buffer frames with progressively increasing noise levels. These buffer frames enable smooth transitions, aligning the fine-tuning process with the pretrained model's temporal dynamics. TIC-FT requires no architectural changes and achieves strong performance with as few as 10-30 training samples. We validate our method across a range of tasks, including image-to-video and video-to-video generation, using large-scale base models such as CogVideoX-5B and Wan-14B. Extensive experiments show that TIC-FT outperforms existing baselines in both condition fidelity and visual quality, while remaining highly efficient in both training and inference. For additional results, visit https://kinam0252.github.io/TIC-FT/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。