arXiv:2508.18633cs.CVcs.AI2025-08NeurIPS被引 30

解决视频去物时阴影反光等副作用的难题,实现更干净的删除效果。

ROSE: Remove Objects with Side Effects in Videos

  • 用3D渲染生成大规模配对数据,模拟五类环境副作用
  • 基于扩散Transformer模型,通过参考帧实现精准去物与副作用消除
  • 自建基准测试集,覆盖真实场景与复杂副作用,适合视频编辑研究者

由于视频生成模型的进展,视频物体移除已取得显著成果。然而,在处理物体带来的副作用(如阴影、反射)时,现有方法因缺乏成对视频数据监督而表现不佳。本文提出ROSE(Remove Objects with Side Effects),系统研究物体对环境的影响,涵盖阴影、反射、光照、半透明和镜面五类常见情况。针对配对数据稀缺问题,我们利用3D渲染引擎构建全自动数据生成流程,模拟多样化场景、物体、视角与相机轨迹,生成大规模配对数据集。ROSE基于扩散变压器实现视频修复,通过整段视频输入进行参考式擦除,并引入额外监督,显式预测受副作用影响区域,该区域可通过配对视频的差异掩码揭示。为全面评估模型性能,我们构建新基准ROSE-Bench,包含常规场景及上述五类特殊副作用。实验表明,ROSE在多个指标上优于现有视频去物模型,并在真实视频中具有良好泛化能力。

原文摘要 · Abstract (English)

Video object removal has achieved advanced performance due to the recent success of video generative models. However, when addressing the side effects of objects, e.g., their shadows and reflections, existing works struggle to eliminate these effects for the scarcity of paired video data as supervision. This paper presents ROSE, termed Remove Objects with Side Effects, a framework that systematically studies the object's effects on environment, which can be categorized into five common cases: shadows, reflections, light, translucency and mirror. Given the challenges of curating paired videos exhibiting the aforementioned effects, we leverage a 3D rendering engine for synthetic data generation. We carefully construct a fully-automatic pipeline for data preparation, which simulates a large-scale paired dataset with diverse scenes, objects, shooting angles, and camera trajectories. ROSE is implemented as an video inpainting model built on diffusion transformer. To localize all object-correlated areas, the entire video is fed into the model for reference-based erasing. Moreover, additional supervision is introduced to explicitly predict the areas affected by side effects, which can be revealed through the differential mask between the paired videos. To fully investigate the model performance on various side effect removal, we presents a new benchmark, dubbed ROSE-Bench, incorporating both common scenarios and the five special side effects for comprehensive evaluation. Experimental results demonstrate that ROSE achieves superior performance compared to existing video object erasing models and generalizes well to real-world video scenarios. The project page is https://rose2025-inpaint.github.io/.

视频去物扩散模型3D渲染图像修复

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。