用图文结合方式生成高质量程序化材质图,更直观高效。
MultiMat: Multimodal Program Synthesis for Procedural Materials using Large Multimodal Models
- 融合视觉与文本信息,利用大模型理解节点图的图形结构
- 在真实生产级数据上训练,生成结果视觉质量更高、结构正确
- 适合需要快速创作3D材质的美术师和设计师使用
材质节点图是生成程序化材质2D通道(如粗糙度、位移、反照率等)的程序,广泛用于计算机图形学中以参数化方式表示虚拟3D物体的外观,并支持任意分辨率。其有向无环图结构与中间状态使工作流具备模块化与可解释性,便于交互式建模。然而,创建这类图仍具挑战性,通常需专业训练。现有神经程序合成方法仅将图表示为文本程序,未能捕捉节点图固有的视觉空间特征,限制了人类可读性。为此,我们提出MultiMat,一个基于大模态模型的多模态程序合成框架,可同时处理视觉与文本形式的节点图表示,提升程序化材质图的生成效果。我们在新构建的生产级程序化材质数据集上训练模型,并结合受限树搜索推理算法,在保证静态正确性的前提下高效探索程序空间。实验表明,该方法在无条件与条件生成任务中均更高效,且生成图像质量与保真度优于纯文本基线,达到新基准性能。
原文摘要 · Abstract (English)
Material node graphs are programs that generate the 2D channels of procedural materials, including geometry such as roughness and displacement maps, and reflectance such as albedo and conductivity maps. They are essential in computer graphics for representing the appearance of virtual 3D objects parametrically and at arbitrary resolution. In particular, their directed acyclic graph structure and intermediate states enable a modular, interpretable workflow for interactive appearance modeling. However, creating such graphs remains challenging and typically requires professional training. While recent neural program synthesis approaches attempt to simplify this process, they solely represent graphs as textual programs, failing to capture the inherently visual-spatial nature of node graphs that makes them accessible to humans. To address this gap, we present MultiMat, a multimodal program synthesis framework that leverages large multimodal models to process both visual and textual graph representations for improved generation of procedural material graphs. We train our models on a new dataset of production-quality procedural materials and combine them with a constrained tree search inference algorithm that ensures static correctness while efficiently navigating the program space. Our experimental results show that our multimodal program synthesis method is more efficient in both unconditional and conditional graph synthesis with higher visual quality and fidelity than text-only baselines, establishing new state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。