无需文本提示，用三维度图像实现高质量材质迁移。
DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer
- 用深度、法线、光照图替代文本控制，简化生成流程。
- 在多种物体和光照下均实现高保真材质迁移，效果稳定。
- 轻量级设计适合实时应用，计算效率高，适合工业级部署。
基于扩散的材质迁移方法通常依赖图像微调或复杂架构中的辅助网络，存在对文本依赖、计算开销大、特征错配等问题。为此，我们提出DealMaTe，利用深度、法线和光照图像实现材质迁移。该方法为简化扩散框架，摒弃文本引导与参考网络。设计轻量级3D信息注入模块——多维3D着色器LoRA，无需修改基础模型权重，即可兼容多种控制条件，并实现协调稳定的生成结果。同时，通过着色器因果互注意机制与键值缓存优化注意力机制，降低多条件带来的推理延迟，提升计算效率，以低复杂度架构实现高质量材质迁移。大量实验在多样物体与光照条件下验证了其鲁棒性，能实现任意输入材质的高保真迁移。代码已开源：https://github.com/haha-lisa/DealMaTe。
原文摘要 · Abstract (English)
Recently, diffusion-based material transfer methods rely on image fine-tuning or complex architectures with auxiliary networks but face challenges such as text dependency, additional computational costs, and feature misalignment. To address these limitations, we propose \textbf{DealMaTe}, using \underline{\textbf{de}}pth, norm\underline{\textbf{a}}l, and \underline{\textbf{l}}ighting images for \underline{\textbf{ma}}terial \underline{\textbf{t}}ransf\underline{\textbf{e}}r. DealMaTe is a simplified diffusion framework that eliminates text guidance and reference networks. We design a lightweight 3D information injection method, Multi-Dim 3D Shader LoRA, which, without modifying the base model weights, enables compatible control conditions and achieves harmonious and stable results. Additionally, we optimize the attention mechanism with Shader Causal Mutual Attention and key-value (KV) caching to reduce inference latency caused by multiple conditions, improve computational efficiency, and achieve high-quality material transfer results with low architectural complexity. Extensive experiments covering a wide variety of objects and lighting conditions consistently demonstrate that DealMaTe achieves remarkable high-fidelity material transfer under arbitrary input materials. The code is available at https://github.com/haha-lisa/DealMaTe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。