用推理步骤实现无需掩码的精准视频编辑
VideoCoF: Unified Video Editing with Temporal Reasoner
- 通过帧链式推理先预测编辑区域，再生成视频
- 仅用5万对数据即达当前最佳性能
- 适合需要精细控制且不想画掩码的用户
现有视频编辑方法面临核心矛盾：专家模型精度高但依赖任务特定先验（如掩码），难以统一；而统一的时序上下文学习模型虽免于掩码，却缺乏显式空间线索，导致指令到区域映射弱、定位不准。为此，我们提出VideoCoF，一种受思维链启发的帧链式方法。该方法强制视频扩散模型遵循‘观察→推理→编辑’流程，先预测推理标记（编辑区域潜在表示），再生成目标视频标记。这一显式推理步骤无需用户提供掩码，同时实现精准的指令-区域对齐与细粒度编辑。此外，我们引入RoPE对齐策略，利用推理标记确保运动一致性，并支持超出训练时长的长度外推。实验表明，仅需5万对视频数据，VideoCoF在VideoCoF-Bench上达到领先性能，验证了方法的高效性与有效性。代码、权重与数据已公开于https://github.com/knightyxp/VideoCoF。
原文摘要 · Abstract (English)
Existing video editing methods face a critical trade-off: expert models offer precision but rely on task-specific priors like masks, hindering unification; conversely, unified temporal in-context learning models are mask-free but lack explicit spatial cues, leading to weak instruction-to-region mapping and imprecise localization. To resolve this conflict, we propose VideoCoF, a novel Chain-of-Frames approach inspired by Chain-of-Thought reasoning. VideoCoF enforces a ``see, reason, then edit" procedure by compelling the video diffusion model to first predict reasoning tokens (edit-region latents) before generating the target video tokens. This explicit reasoning step removes the need for user-provided masks while achieving precise instruction-to-region alignment and fine-grained video editing. Furthermore, we introduce a RoPE alignment strategy that leverages these reasoning tokens to ensure motion alignment and enable length extrapolation beyond the training duration. We demonstrate that with a minimal data cost of only 50k video pairs, VideoCoF achieves state-of-the-art performance on VideoCoF-Bench, validating the efficiency and effectiveness of our approach. Our code, weight, data are available at https://github.com/knightyxp/VideoCoF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。