用扩散模型生成更真实一致的3D材质，解决多视角不一致和泛化差问题。
MCMat: Multiview-Consistent and Physically Accurate PBR Material Generation
- 分两阶段生成：先用DiT生成多视角材质，再在UV空间细化细节。
- 引入参考注意力机制和物理损失，使材质在不同视角下保持一致性。
- 适合需要高保真材质的3D建模、游戏开发与图形重光照场景。
现有2D方法使用基于UNet的扩散模型生成多视角物理渲染（PBR）贴图，但存在多视角不一致问题；部分3D方法直接生成UV贴图，因3D数据有限导致泛化能力差。为此，我们提出两阶段方法：第一阶段采用扩散Transformer（DiT）生成PBR材质，通过多分支DiT和基于参考的DiT块中的全局注意力机制，促进跨视角特征交互与融合，提升一致性；同时引入基于PBR的扩散损失，确保生成材质符合物理规律。第二阶段提出材质精炼DiT，对空缺区域进行修复并增强细节，以第一阶段输出的材质图为附加条件，降低学习难度、提升泛化能力。大量实验表明，该方法在3D物体纹理化任务中达到当前最优性能，并显著提升图形重光照效果。
原文摘要 · Abstract (English)
Existing 2D methods utilize UNet-based diffusion models to generate multi-view physically-based rendering (PBR) maps but struggle with multi-view inconsistency, while some 3D methods directly generate UV maps, encountering generalization issues due to the limited 3D data. To address these problems, we propose a two-stage approach, including multi-view generation and UV materials refinement. In the generation stage, we adopt a Diffusion Transformer (DiT) model to generate PBR materials, where both the specially designed multi-branch DiT and reference-based DiT blocks adopt a global attention mechanism to promote feature interaction and fusion between different views, thereby improving multi-view consistency. In addition, we adopt a PBR-based diffusion loss to ensure that the generated materials align with realistic physical principles. In the refinement stage, we propose a material-refined DiT that performs inpainting in empty areas and enhances details in UV space. Except for the normal condition, this refinement also takes the material map from the generation stage as an additional condition to reduce the learning difficulty and improve generalization. Extensive experiments show that our method achieves state-of-the-art performance in texturing 3D objects with PBR materials and provides significant advantages for graphics relighting applications. Project Page: https://lingtengqiu.github.io/2024/MCMat/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。