用微调扩散模型一步完成视频抠像,速度快质量高。
EasyOmnimatte: Taming Pretrained Inpainting Diffusion Models for End-to-End Video Layered Decomposition
- 用双专家机制微调预训练视频修复模型,分离前景与影响区域
- 在早期噪声阶段用效果专家捕捉整体结构,后期由质量专家优化细节
- 无需多次采样,计算成本低,适合实际应用
现有视频全向蒙版方法通常依赖缓慢的多阶段或推理时优化流程,未能充分利用强大的生成先验,导致分解效果不佳。本文核心洞察是:若视频修复模型能通过微调消除前景相关效应,则其必然具备感知这些效应的能力,因此也可用于互补任务——前景层及其影响区域的分解。然而,直接对所有模块应用LoRA微调虽能生成高质量透明度图,却无法捕捉关联效应。系统分析表明,这是由于效应相关线索主要编码在特定DiT块中,而全模块应用LoRA会抑制这些特征。为此,我们提出EasyOmnimatte,首个统一、端到端的视频全向蒙版方法。具体而言,我们在保持原始权重不变的前提下,微调预训练视频修复扩散模型以学习两个互补专家:仅在敏感块上应用LoRA的‘效果专家’,用于捕捉前景及影响区域的粗略结构;以及全模块微调的‘质量专家’,负责精细化透明度图。采样过程中,早期高噪声阶段使用效果专家去噪,后期低噪声阶段切换至质量专家。该设计避免了两次完整扩散过程,显著降低计算成本,同时保证输出质量。消融实验验证了双专家策略的有效性。实验表明,EasyOmnimatte在视频全向蒙版任务上达到新基准,并显著优于基线,在质量和效率上均有提升。
原文摘要 · Abstract (English)
Existing video omnimatte methods typically rely on slow, multi-stage, or inference-time optimization pipelines that fail to fully exploit powerful generative priors, producing suboptimal decompositions. Our key insight is that, if a video inpainting model can be finetuned to remove the foreground-associated effects, then it must be inherently capable of perceiving these effects, and hence can also be finetuned for the complementary task: foreground layer decomposition with associated effects. However, although naïvely finetuning the inpainting model with LoRA applied to all blocks can produce high-quality alpha mattes, it fails to capture associated effects. Our systematic analysis reveals this arises because effect-related cues are primarily encoded in specific DiT blocks and become suppressed when LoRA is applied across all blocks. To address this, we introduce EasyOmnimatte, the first unified, end-to-end video omnimatte method. Concretely, we finetune a pretrained video inpainting diffusion model to learn dual complementary experts while keeping its original weights intact: an Effect Expert, where LoRA is applied only to effect-sensitive DiT blocks to capture the coarse structure of the foreground and associated effects, and a fully LoRA-finetuned Quality Expert learns to refine the alpha matte. During sampling, Effect Expert is used for denoising at early, high-noise steps, while Quality Expert takes over at later, low-noise steps. This design eliminates the need for two full diffusion passes, significantly reducing computational cost without compromising output quality. Ablation studies validate the effectiveness of this Dual-Expert strategy. Experiments demonstrate that EasyOmnimatte sets a new state-of-the-art for video omnimatte and enables various downstream tasks, significantly outperforming baselines in both quality and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。