用跨图注意力修复图像超分模型在材质生成中的不一致问题
MUJICA: Reforming SISR Models for PBR Material Super-Resolution via Cross-Map Attention
- 引入跨图注意力机制融合多纹理图特征
- 在多个数据集上提升PSNR、SSIM，保持材质间一致性
- 适配现有模型，低资源下仍达顶尖性能
基于物理的渲染（PBR）材质通常由多张2D贴图（如基础色、法线、金属度、粗糙度）构成，编码空间变化的双向反射分布函数（SVBRDF）参数以建模表面反射特性。对这些材质进行超分辨率重建对现代3D图形应用具有重要意义。然而，现有单图像超分辨率（SISR）方法在跨贴图一致性、模态特异性特征建模以及数据分布偏移下的泛化能力方面存在不足。本文提出多模态联合上采样跨图注意力（MUJICA），一种可灵活接入预训练的Swin Transformer-based SISR模型的适配器。该模块在冻结的SISR主干后无缝嵌入，通过跨图注意力融合特征，同时保留原模型的优秀重建能力。在SwinIR、DRCT和HMANet等模型上应用，MUJICA在保持跨贴图一致性的同时显著提升PSNR、SSIM和LPIPS指标。实验表明，该方法在资源受限条件下仍能高效训练，并在PBR材质数据集上达到当前最优性能。
原文摘要 · Abstract (English)
Physically Based Rendering (PBR) materials are typically characterized by multiple 2D texture maps such as basecolor, normal, metallic, and roughness which encode spatially-varying bi-directional reflectance distribution function (SVBRDF) parameters to model surface reflectance properties and microfacet interactions. Upscaling SVBRDF material is valuable for modern 3D graphics applications. However, existing Single Image Super-Resolution (SISR) methods struggle with cross-map inconsistency, inadequate modeling of modality-specific features, and limited generalization due to data distribution shifts. In this work, we propose Multi-modal Upscaling Joint Inference via Cross-map Attention (MUJICA), a flexible adapter that reforms pre-trained Swin-transformer-based SISR models for PBR material super-resolution. MUJICA is seamlessly attached after the pre-trained and frozen SISR backbone. It leverages cross-map attention to fuse features while preserving remarkable reconstruction ability of the pre-trained SISR model. Applied to SISR models such as SwinIR, DRCT, and HMANet, MUJICA improves PSNR, SSIM, and LPIPS scores while preserving cross-map consistency. Experiments demonstrate that MUJICA enables efficient training even with limited resources and delivers state-of-the-art performance on PBR material datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。