让扩散模型生成更真实的镜面反射,解决物体位置和遮挡问题。
MirrorVerse: Pushing Diffusion Models to Realistically Reflect the World
- 用随机位置、旋转和物体定位增强合成数据,提升泛化能力。
- 构建配对物体数据集,有效处理多物体场景中的空间关系与遮挡。
- 三阶段训练策略使模型在真实场景中表现更优,适合图像编辑应用。
扩散模型在图像编辑中占据核心地位,但在阴影、反射和遮挡等物理规律遵循方面表现不佳。本文针对生成逼真镜面反射的问题,提出改进方法。尽管已有大量训练数据,现有模型仍难以捕捉镜面反射的细微特征。近期方法尝试通过合成数据并将其视为修复任务来解决,但泛化能力受限于物体相对于镜子的位置和朝向。为此,本文在合成数据生成中引入三项关键增强:(1) 随机物体位置,(2) 随机旋转,(3) 物体定位约束,显著提升对不同姿态和摆放的泛化能力。为进一步处理多物体场景中的空间关系与遮挡,我们设计了物体配对策略生成数据集,使其具备更强鲁棒性。为提升真实场景表现,采用三阶段训练课程训练出 MirrorFusion 2.0 模型。通过大量定性和定量评估验证方法有效性。项目主页见:https://mirror-verse.github.io/。
原文摘要 · Abstract (English)
Diffusion models have become central to various image editing tasks, yet they often fail to fully adhere to physical laws, particularly with effects like shadows, reflections, and occlusions. In this work, we address the challenge of generating photorealistic mirror reflections using diffusion-based generative models. Despite extensive training data, existing diffusion models frequently overlook the nuanced details crucial to authentic mirror reflections. Recent approaches have attempted to resolve this by creating synhetic datasets and framing reflection generation as an inpainting task; however, they struggle to generalize across different object orientations and positions relative to the mirror. Our method overcomes these limitations by introducing key augmentations into the synthetic data pipeline: (1) random object positioning, (2) randomized rotations, and (3) grounding of objects, significantly enhancing generalization across poses and placements. To further address spatial relationships and occlusions in scenes with multiple objects, we implement a strategy to pair objects during dataset generation, resulting in a dataset robust enough to handle these complex scenarios. Achieving generalization to real-world scenes remains a challenge, so we introduce a three-stage training curriculum to develop the MirrorFusion 2.0 model to improve real-world performance. We provide extensive qualitative and quantitative evaluations to support our approach. The project page is available at: https://mirror-verse.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。