用3D资产作参考生成图像，实现2D与3D属性精准对齐。
RefAny3D: 3D Asset-Referenced Diffusion Models for Image Generation
- 双分支架构融合多视角图像与点云，联合建模颜色和空间坐标
- 生成的图像与3D参考保持空间一致性，内容可解耦
- 适合3D内容创作、数字孪生等需要精确几何对齐的场景
本文提出一种基于3D资产的扩散模型，用于图像生成。现有基于参考的图像生成方法依赖大规模预训练扩散模型，能以单张图像为参考生成多样化图像，但仅限于2D参考，无法利用3D资产，限制了实用性。为弥补这一缺口，我们设计了一种跨域双分支感知扩散模型，利用3D资产的多视角RGB图像和点图，联合建模其颜色与规范空间坐标，实现生成图像与3D参考的精确一致性。所提出的空间对齐双分支生成架构与领域解耦生成机制，可同时生成一对空间对齐但内容解耦的输出：RGB图像与点图，将2D图像属性与3D资产属性关联。实验表明，该方法能有效以3D资产为参考生成一致的图像，为扩散模型与3D内容创作的结合开辟新路径。
原文摘要 · Abstract (English)
In this paper, we propose a 3D asset-referenced diffusion model for image generation, exploring how to integrate 3D assets into image diffusion models. Existing reference-based image generation methods leverage large-scale pretrained diffusion models and demonstrate strong capability in generating diverse images conditioned on a single reference image. However, these methods are limited to single-image references and cannot leverage 3D assets, constraining their practical versatility. To address this gap, we present a cross-domain diffusion model with dual-branch perception that leverages multi-view RGB images and point maps of 3D assets to jointly model their colors and canonical-space coordinates, achieving precise consistency between generated images and the 3D references. Our spatially aligned dual-branch generation architecture and domain-decoupled generation mechanism ensure the simultaneous generation of two spatially aligned but content-disentangled outputs, RGB images and point maps, linking 2D image attributes with 3D asset attributes. Experiments show that our approach effectively uses 3D assets as references to produce images consistent with the given assets, opening new possibilities for combining diffusion models with 3D content creation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。