用VR手绘草图直接生成高质量3D模型，支持多模态控制。
VRsketch2Gaussian: 3D VR Sketch Guided 3D Object Generation with Gaussian Splatting
- 通过两阶段对齐，将稀疏的VR草图与CLIP特征对齐
- 用草图控制形状、文字控制外观，实现精细条件生成
- 基于3D高斯泼溅，快速生成纹理丰富的原生3D模型
我们提出VRSketch2Gaussian，首个基于VR草图引导的多模态原生3D物体生成框架，采用3D高斯泼溅（3DGS）表示。为此，我们构建了首个大规模配对数据集VRSS，包含VR草图、文本、图像和3DGS，填补了多模态VR草图生成的空白。方法创新包括：1）草图-CLIP特征对齐，通过两阶段策略弥合稀疏VR草图嵌入与丰富CLIP嵌入之间的领域差异，支持草图检索与生成任务；2）细粒度多模态条件控制，分离3D生成过程，使用显式VR草图进行几何建模，文本描述控制外观；为此设计通用的VR草图编码器以实现跨模态对齐；3）高效高保真原生3D生成，采用原生3D生成范式，实现快速且纹理丰富的3D物体合成。在自建的VRSS数据集上的实验表明，该方法可实现高质量多模态VR草图驱动的3D生成。我们认为该数据集与方法将推动3D生成领域发展。
原文摘要 · Abstract (English)
We propose VRSketch2Gaussian, a first VR sketch-guided, multi-modal, native 3D object generation framework that incorporates a 3D Gaussian Splatting representation. As part of our work, we introduce VRSS, the first large-scale paired dataset containing VR sketches, text, images, and 3DGS, bridging the gap in multi-modal VR sketch-based generation. Our approach features the following key innovations: 1) Sketch-CLIP feature alignment. We propose a two-stage alignment strategy that bridges the domain gap between sparse VR sketch embeddings and rich CLIP embeddings, facilitating both VR sketch-based retrieval and generation tasks. 2) Fine-Grained multi-modal conditioning. We disentangle the 3D generation process by using explicit VR sketches for geometric conditioning and text descriptions for appearance control. To facilitate this, we propose a generalizable VR sketch encoder that effectively aligns different modalities. 3) Efficient and high-fidelity 3D native generation. Our method leverages a 3D-native generation approach that enables fast and texture-rich 3D object synthesis. Experiments conducted on our VRSS dataset demonstrate that our method achieves high-quality, multi-modal VR sketch-based 3D generation. We believe our VRSS dataset and VRsketch2Gaussian method will be beneficial for the 3D generation community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。