用掩码引导的LoRA微调,实现视频编辑的精细时序控制。
LoRA-Edit: Controllable First-Frame-Guided Video Editing via Mask-Aware LoRA Fine-Tuning
- 通过时空掩码指导LoRA微调,让模型学会保留或生成特定区域内容。
- 可同时保持原视频运动一致性或引入参考帧的新外观变化。
- 适合需要精准控制视频演化过程的创作者和研究人员。
基于扩散模型的视频编辑已实现高质量生成效果,但现有方法多依赖大规模预训练,灵活性不足。首帧引导编辑虽能控制初始帧,却难以精细调控后续时序演化。为此,我们提出一种基于掩码的LoRA(低秩自适应)微调方法,用于灵活调整预训练图像到视频模型。核心创新在于使用时空掩码,战略性引导微调过程。该方法使模型掌握双重能力:一是将掩码视为指令,决定在指定区域保留源视频内容或生成新内容;二是对生成区域,学习继承视频原有运动的一致性,或根据用户提供的参考帧生成新颖外观。这种双能力机制赋予用户对编辑全过程的时序控制力,支持如物体旋转、花朵绽放等复杂变换。实验表明,本方法在视频编辑性能上优于基线方法。代码与视频结果详见项目网站:https://cjeen.github.io/LoRAEdit。
原文摘要 · Abstract (English)
Video editing using diffusion models has achieved remarkable results in generating high-quality edits for videos. However, current methods often rely on large-scale pretraining, limiting flexibility for specific edits. First-frame-guided editing provides control over the first frame, but lacks fine-grained control over the edit's subsequent temporal evolution. To address this, we propose a mask-based LoRA (Low-Rank Adaptation) tuning method that adapts pretrained Image-to-Video models for flexible video editing. Our key innovation is using a spatiotemporal mask to strategically guide the LoRA fine-tuning process. This teaches the model two distinct skills: first, to interpret the mask as a command to either preserve content from the source video or generate new content in designated regions. Second, for these generated regions, LoRA learns to synthesize either temporally consistent motion inherited from the video or novel appearances guided by user-provided reference frames. This dual-capability LoRA grants users control over the edit's entire temporal evolution, allowing complex transformations like an object rotating or a flower blooming. Experimental results show our method achieves superior video editing performance compared to baseline methods. The code and video results are available at our project website: https://cjeen.github.io/LoRAEdit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。