通过解耦语义与运动建模,实现精准编辑且保持动作自然。
SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing
- 将视频编辑分解为语义锚定和运动对齐两阶段处理。
- 预训练阶段仅用原始视频就实现零样本编辑,效果接近商用系统。
- 适合需要高保真动作还原的视频编辑场景,如影视剪辑。
当前指令引导的视频编辑模型难以同时实现精确语义修改与动作忠实性。现有方法依赖外部先验(如视觉语言模型特征),严重制约模型鲁棒性。为此,我们提出SAMA(因子化语义锚定与运动对齐)框架,将视频编辑解耦为语义锚定与运动建模。首先,语义锚定在稀疏锚点帧上联合预测语义标记与视频潜在表示,实现纯指令感知的结构规划。其次,运动对齐在运动中心的视频修复任务(立方体补全、速度扰动、管状打乱)上预训练同一骨干网络,使模型直接从原始视频中内化时序动态。SAMA采用两阶段流程:无配对数据的因子化预训练,后接有监督微调。令人惊讶的是,仅预训练阶段即具备强大零样本编辑能力,验证了该解耦设计的有效性。SAMA在开源模型中达到领先性能,媲美顶级商业系统(如Kling-Omni)。代码、模型与数据集将公开。
原文摘要 · Abstract (English)
Current instruction-guided video editing models struggle to simultaneously balance precise semantic modifications with faithful motion preservation. While existing approaches rely on injecting explicit external priors (e.g., VLM features or structural conditions) to mitigate these issues, this reliance severely bottlenecks model robustness and generalization. To overcome this limitation, we present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling. First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames, enabling purely instruction-aware structural planning. Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks (cube inpainting, speed perturbation, and tube shuffle), enabling the model to internalize temporal dynamics directly from raw videos. SAMA is optimized with a two-stage pipeline: a factorized pre-training stage that learns inherent semantic-motion representations without paired video-instruction editing data, followed by supervised fine-tuning on paired editing data. Remarkably, the factorized pre-training alone already yields strong zero-shot video editing ability, validating the proposed factorization. SAMA achieves state-of-the-art performance among open-source models and is competitive with leading commercial systems (e.g., Kling-Omni). Code, models, and datasets will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。