通过拖拽种子点实现3D高斯点云的几何一致生成与编辑
Dragen3D: Multiview Geometry Consistent 3D Gaussian Generation with Drag-Based Control
- 用锚点潜变量编码图像和点云,实现高效3D高斯生成
- 通过稀疏种子点驱动,确保多视角几何一致性
- 无需2D扩散模型,支持直观交互式3D编辑,适合内容创作者
单图像3D生成在虚拟现实、3D建模和数字内容创作中日益重要。然而,现有方法普遍存在多视角几何不一致和生成过程可控性差的问题。为此,我们提出Dragen3D,一种基于3D高斯溅射(3DGS)的新型方法。引入锚点高斯变分自编码器(Anchor-GS VAE),将点云与单张图像编码为锚点潜变量,并解码生成3DGS。为实现多视角几何一致且可控制的生成,提出种子点驱动策略:先生成稀疏种子点作为粗略几何表示,再通过种子-锚点映射模块将其转换为锚点潜变量。几何一致性由易于学习的稀疏种子点保证,用户可直观拖拽种子点以变形最终3DGS,变化通过锚点潜变量传播。据我们所知,这是首个无需依赖2D扩散先验即可实现几何可控3D高斯生成与编辑的方法,生成质量媲美当前最先进水平。
原文摘要 · Abstract (English)
Single-image 3D generation has emerged as a prominent research topic, playing a vital role in virtual reality, 3D modeling, and digital content creation. However, existing methods face challenges such as a lack of multi-view geometric consistency and limited controllability during the generation process, which significantly restrict their usability. % To tackle these challenges, we introduce Dragen3D, a novel approach that achieves geometrically consistent and controllable 3D generation leveraging 3D Gaussian Splatting (3DGS). We introduce the Anchor-Gaussian Variational Autoencoder (Anchor-GS VAE), which encodes a point cloud and a single image into anchor latents and decode these latents into 3DGS, enabling efficient latent-space generation. To enable multi-view geometry consistent and controllable generation, we propose a Seed-Point-Driven strategy: first generate sparse seed points as a coarse geometry representation, then map them to anchor latents via the Seed-Anchor Mapping Module. Geometric consistency is ensured by the easily learned sparse seed points, and users can intuitively drag the seed points to deform the final 3DGS geometry, with changes propagated through the anchor latents. To the best of our knowledge, we are the first to achieve geometrically controllable 3D Gaussian generation and editing without relying on 2D diffusion priors, delivering comparable 3D generation quality to state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。