用扩散模型生成逼真镜面反射,支持用户控制镜子位置。
Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections
- 将镜面反射建模为图像修复任务,提升生成可控性。
- 在19.8万样本数据集上,生成效果优于现有方法。
- 适合图像编辑与增强现实研究者使用。
本文针对基于扩散模型生成高度真实且合理的镜面反射问题提出解决方案。将该问题定义为图像修复任务,从而在生成过程中实现对镜子位置的更多用户控制。为此,构建了大规模合成数据集SynMirror,包含约19.8万张由6.6万个唯一3D物体渲染的图像,附带深度图、法线图和实例分割掩码,以捕捉场景的几何特性。基于此数据集,提出一种新的深度条件图像修复方法MirrorFusion,能够生成高质量、形状与外观感知的镜面反射。大量定量与定性分析表明,MirrorFusion在SynMirror数据集上的表现超越现有最先进方法。据我们所知,这是首次成功利用扩散模型实现对场景中物体镜面反射的可控且忠实生成。SynMirror与MirrorFusion为图像编辑与增强现实应用开辟了新路径。
原文摘要 · Abstract (English)
We tackle the problem of generating highly realistic and plausible mirror reflections using diffusion-based generative models. We formulate this problem as an image inpainting task, allowing for more user control over the placement of mirrors during the generation process. To enable this, we create SynMirror, a large-scale dataset of diverse synthetic scenes with objects placed in front of mirrors. SynMirror contains around 198k samples rendered from 66k unique 3D objects, along with their associated depth maps, normal maps and instance-wise segmentation masks, to capture relevant geometric properties of the scene. Using this dataset, we propose a novel depth-conditioned inpainting method called MirrorFusion, which generates high-quality, realistic, shape and appearance-aware reflections of real-world objects. MirrorFusion outperforms state-of-the-art methods on SynMirror, as demonstrated by extensive quantitative and qualitative analysis. To the best of our knowledge, we are the first to successfully tackle the challenging problem of generating controlled and faithful mirror reflections of an object in a scene using diffusion-based models. SynMirror and MirrorFusion open up new avenues for image editing and augmented reality applications for practitioners and researchers alike. The project page is available at: https://val.cds.iisc.ac.in/reflecting-reality.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。