零样本视频背景替换，保持主体一致且光照自然。
ANYPORTAL: Zero-Shot Consistent Video Background Replacement
- 融合视频与图像扩散模型，零样本实现背景替换。
- 提出精修投影算法，确保前景像素级一致性。
- 无需训练，在普通显卡上高效运行，适合内容创作者。
尽管视频生成技术快速进步，但精确匹配用户意图的高质量视频创作仍面临挑战。现有方法往往难以实现对视频细节的精细控制，限制了实际应用。我们提出 ANYPORTAL，一种基于预训练扩散模型的零样本视频背景替换框架。该框架协同利用视频扩散模型的时间先验与图像扩散模型的光影重映射能力，在零样本设置下完成背景替换。针对前景一致性这一关键难题，我们提出精修投影算法，实现像素级细节操控，确保前景精准保留。ANYPORTAL 不需训练，克服了前景一致性与时间连贯性光照重映射的挑战。实验表明，ANYPORTAL 在消费级 GPU 上即可实现高质量结果，为视频内容创作与编辑提供高效实用的解决方案。
原文摘要 · Abstract (English)
Despite the rapid advancements in video generation technology, creating high-quality videos that precisely align with user intentions remains a significant challenge. Existing methods often fail to achieve fine-grained control over video details, limiting their practical applicability. We introduce ANYPORTAL, a novel zero-shot framework for video background replacement that leverages pre-trained diffusion models. Our framework collaboratively integrates the temporal prior of video diffusion models with the relighting capabilities of image diffusion models in a zero-shot setting. To address the critical challenge of foreground consistency, we propose a Refinement Projection Algorithm, which enables pixel-level detail manipulation to ensure precise foreground preservation. ANYPORTAL is training-free and overcomes the challenges of achieving foreground consistency and temporally coherent relighting. Experimental results demonstrate that ANYPORTAL achieves high-quality results on consumer-grade GPUs, offering a practical and efficient solution for video content creation and editing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。