通过CLIP空间嵌入实现材质混合与精细控制，支持单次前向传播多编辑。
MARBLE: Material Recomposition and Blending in CLIP-Space
- 在CLIP空间中寻找材质嵌入，用以控制预训练文生图模型。
- 可实现粗糙度、金属感等细粒度材质属性的参数化调控。
- 支持多材质融合与绘画应用，单次推理完成多个编辑操作。
基于样本图像的图像物体材质编辑是计算机视觉与图形学中的研究热点。我们提出MARBLE，一种在CLIP空间中寻找材质嵌入并利用其控制预训练文生图模型的方法，实现材质混合与细粒度材质重构。通过定位去噪UNet中负责材质归属的模块，我们能够基于两个材质样本图像，在CLIP空间中找到材质混合方向。此外，通过一个浅层网络预测目标材质属性（如粗糙度、金属感、透明度、发光）变化的方向，实现对细粒度材质属性的参数化控制。我们通过定性和定量分析验证了方法的有效性，并展示了单次前向传播中完成多次编辑的能力以及在绘画中的适用性。项目页面：https://marblecontrol.github.io/
原文摘要 · Abstract (English)
Editing materials of objects in images based on exemplar images is an active area of research in computer vision and graphics. We propose MARBLE, a method for performing material blending and recomposing fine-grained material properties by finding material embeddings in CLIP-space and using that to control pre-trained text-to-image models. We improve exemplar-based material editing by finding a block in the denoising UNet responsible for material attribution. Given two material exemplar-images, we find directions in the CLIP-space for blending the materials. Further, we can achieve parametric control over fine-grained material attributes such as roughness, metallic, transparency, and glow using a shallow network to predict the direction for the desired material attribute change. We perform qualitative and quantitative analysis to demonstrate the efficacy of our proposed method. We also present the ability of our method to perform multiple edits in a single forward pass and applicability to painting. Project Page: https://marblecontrol.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。