无需3D信息，一键实现照片中材质的逼真替换
MatSwap: Light-aware material transfers in images
- 基于光照与几何感知的扩散模型，直接学习材质在场景中的表现
- 在真实图像上实现高保真材质替换，保持原场景结构不变
- 适用于普通用户，无需专业标注或3D建模知识
我们提出MatSwap，一种在图像中指定区域光感逼真地转移材质的方法。由于材质外观、几何形状和光照在照片中高度耦合，该任务极具挑战性。现有方法通常依赖繁琐的文字描述或需艺术家参与的手动标注，且依赖难以获取的3D场景信息。相比之下，我们直接学习输入材质（在平面表面观察到）与其在真实场景中的外观之间的关系，无需显式UV映射。为此，我们构建了一个定制的光照与几何感知扩散模型，并使用合成数据微调大规模预训练文本到图像模型，保留其强先验以确保对真实图像的有效泛化。结果表明，该方法能无缝将目标材质融入照片中特定位置，同时保持场景原有身份。我们在合成与真实图像上进行评估，结果表明其在定性和定量上均优于近期方法。代码与数据已开源：https://github.com/astra-vision/MatSwap。
原文摘要 · Abstract (English)
We present MatSwap, a method to transfer materials to designated surfaces in an image photorealistically. Such a task is non-trivial due to the large entanglement of material appearance, geometry, and lighting in a photograph. In the literature, material editing methods typically rely on either cumbersome text engineering or extensive manual annotations requiring artist knowledge and 3D scene properties that are impractical to obtain. In contrast, we propose to directly learn the relationship between the input material -- as observed on a flat surface -- and its appearance within the scene, without the need for explicit UV mapping. To achieve this, we rely on a custom light- and geometry-aware diffusion model. We fine-tune a large-scale pre-trained text-to-image model for material transfer using our synthetic dataset, preserving its strong priors to ensure effective generalization to real images. As a result, our method seamlessly integrates a desired material into the target location in the photograph while retaining the identity of the scene. We evaluate our method on synthetic and real images and show that it compares favorably to recent work both qualitatively and quantitatively. We release our code and data on https://github.com/astra-vision/MatSwap
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。