用扩散模型实现可控制的3D渲染，让生成图像更真实可控。
RGBX-Next: Towards Realistic Generative Rendering from G-Buffers
- 将扩散变换器微调为正向与逆向渲染器，基于G-buffers生成图像。
- 在真实渲染和内部属性分解上均达到高质量效果。
- 适合关注可控生成与3D内容创作的研究者和开发者。
扩散模型在图像、视频和流媒体生成中取得了显著进展，但相比传统3D渲染，仍缺乏对生成结果的精确控制。我们认为可行的前进方向是将生成模型作为基于传统渲染生成缓冲区（G-buffers）的条件化渲染器。本文提出RGBX-Next，一个统一的生成框架，支持正向与逆向渲染：既能从图像、视频和流中估计G-buffers，也能从G-buffers中渲染出真实感图像、视频和流。核心贡献是为微调扩散变压器（DiT）模型成为生成式正向与逆向渲染器提供通用方法。实验表明，所获模型在真实感生成与内在属性分解任务上表现优异。所有模型将公开发布。我们相信本文提出的设计原则将推动未来可控生成式正向与逆向渲染研究。
原文摘要 · Abstract (English)
Diffusion models have achieved impressive results in image, video, and streaming generation. However, compared to traditional 3D rendering, they still lack precise control over the generated output. We believe a viable path forward is to use generative models as learned renderers conditioned on traditionally rendered G-buffers. We introduce RGBX-Next, a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images, videos, and streams, and rendering realistic images, videos, and streams from G-buffers. Our key contribution is a general recipe for finetuning diffusion transformer (DiT) models into generative forward and inverse renderers. We show that the resulting models achieve high quality in both realistic generative rendering and intrinsic decomposition. We will make all our models publicly available. We believe that the design principles presented in this paper will benefit future research on controllable generative forward and inverse rendering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。