用引导流实现3D物体外观迁移,解决几何差异下的失真问题。
GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer
- 基于预训练流模型,通过周期性引导优化采样过程。
- 在不同几何下仍能保留纹理与细节,优于现有方法。
- 适合游戏、AR等需高质量外观迁移的场景。
利用图像或文本等外观对象表示进行3D资产外观迁移,已在游戏、增强现实和数字内容创作等领域引发关注。然而,当输入与外观对象几何差异显著时,现有方法仍表现不佳。直接应用3D生成模型虽简单,但难以生成令人满意的外观结果。为此,我们提出一种受通用引导启发的系统性方法:给定条件为图像或文本的预训练修正流模型,本方法无需训练,在采样过程中周期性加入引导。该引导可建模为可微分损失函数,实验中采用两种引导形式——面向部件的外观损失与自相似性损失。实验表明,该方法成功将纹理与几何细节迁移到输入3D资产上,无论定性还是定量均优于基线。同时发现,传统评价指标因无法聚焦局部细节且难以比较异构输入(无真实标签),故不适用。因此,我们采用基于GPT的系统客观排序输出,确保评估具有人类判断一致性,用户研究进一步验证了其有效性。本方法具通用性,可扩展至多种扩散模型与引导函数。
原文摘要 · Abstract (English)
Transferring appearance to 3D assets using different representations of the appearance object - such as images or text - has garnered interest due to its wide range of applications in industries like gaming, augmented reality, and digital content creation. However, state-of-the-art methods still fail when the geometry between the input and appearance objects is significantly different. A straightforward approach is to directly apply a 3D generative model, but we show that this ultimately fails to produce appealing results. Instead, we propose a principled approach inspired by universal guidance. Given a pretrained rectified flow model conditioned on image or text, our training-free method interacts with the sampling process by periodically adding guidance. This guidance can be modeled as a differentiable loss function, and we experiment with two different types of guidance including part-aware losses for appearance and self-similarity. Our experiments show that our approach successfully transfers texture and geometric details to the input 3D asset, outperforming baselines both qualitatively and quantitatively. We also show that traditional metrics are not suitable for evaluating the task due to their inability of focusing on local details and comparing dissimilar inputs, in absence of ground truth data. We thus evaluate appearance transfer quality with a GPT-based system objectively ranking outputs, ensuring robust and human-like assessment, as further confirmed by our user study. Beyond showcased scenarios, our method is general and could be extended to different types of diffusion models and guidance functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。