将参考物体的局部纹理精准迁移至目标物体,实现可控图像视频编辑。
Refaçade: Editing Object with Given Reference Texture
- 用纹理移除器剥离源物体外观,保留几何与运动信息。
- 通过拼图打乱参考图全局布局,聚焦局部纹理统计特征。
- 在图像与视频上实现高精度、可控制的纹理编辑,优于现有方法。
扩散模型的进展推动了图像与视频编辑的突破,但部分任务仍待探索。本文提出新任务「物体重纹理(Object Retexture)」,即在图像或视频中将参考物体的局部纹理迁移到目标物体。直接使用基于源结构与参考纹理的ControlNet存在可控性不足问题:原始参考图引入无关结构信息,且未能解耦源物体的纹理与结构。为此,我们提出Refaçade,包含两项核心设计:首先,采用在成对有/无纹理3D网格渲染数据上训练的纹理移除器,去除外观信息同时保留几何与运动;其次,通过拼图置换打乱参考图全局布局,引导模型关注局部纹理统计而非整体构型。大量实验表明,该方法在视觉质量、编辑精度与可控性方面均优于强基线,在定量与人工评估中表现更优。代码已开源:https://github.com/fishZe233/Refacade。
原文摘要 · Abstract (English)
Recent advances in diffusion models have brought remarkable progress in image and video editing, yet some tasks remain underexplored. In this paper, we introduce a new task, Object Retexture, which transfers local textures from a reference object to a target object in images or videos. To perform this task, a straightforward solution is to use ControlNet conditioned on the source structure and the reference texture. However, this approach suffers from limited controllability for two reasons: conditioning on the raw reference image introduces unwanted structural information, and it fails to disentangle the visual texture and structure information of the source. To address this problem, we propose Refaçade, a method that consists of two key designs to achieve precise and controllable texture transfer in both images and videos. First, we employ a texture remover trained on paired textured/untextured 3D mesh renderings to remove appearance information while preserving the geometry and motion of source videos. Second, we disrupt the reference global layout using a jigsaw permutation, encouraging the model to focus on local texture statistics rather than the global layout of the object. Extensive experiments demonstrate superior visual quality, precise editing, and controllability, outperforming strong baselines in both quantitative and human evaluations. Code is available at https://github.com/fishZe233/Refacade.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。