用多视角生成重建先验,实现灵活的3D拖拽编辑。
MvDrag3D: Drag-based Creative 3D Editing via Multi-view Generation-Reconstruction Priors
- 通过多视角扩散模型提供生成先验,实现跨视角一致编辑。
- 支持复杂拓扑变化和新纹理生成,适配多种物体类别。
- 适合需要创意3D内容生成的设计师与开发者使用。
拖拽编辑在2D内容创作中已广泛应用,但拓展至3D仍面临挑战。现有方法在处理显著拓扑变化或跨类别新纹理生成时表现不足。为此,我们提出MvDrag3D框架,利用多视角生成与重建先验实现更灵活的3D拖拽编辑。核心是采用多视角扩散模型作为强生成先验,在多个渲染视图上进行一致性拖拽编辑,并通过重建模型恢复编辑后对象的3D Gaussians。针对不同视图间Gaussians可能存在的错位问题,引入视图特异性形变网络进行位置调整。此外,设计多视角得分函数,从多视角中提炼生成先验,进一步提升视图一致性与视觉质量。大量实验表明,MvDrag3D提供了精确、生成式且灵活的3D拖拽编辑方案,适用于多种物体类别和3D表示形式。
原文摘要 · Abstract (English)
Drag-based editing has become popular in 2D content creation, driven by the capabilities of image generative models. However, extending this technique to 3D remains a challenge. Existing 3D drag-based editing methods, whether employing explicit spatial transformations or relying on implicit latent optimization within limited-capacity 3D generative models, fall short in handling significant topology changes or generating new textures across diverse object categories. To overcome these limitations, we introduce MVDrag3D, a novel framework for more flexible and creative drag-based 3D editing that leverages multi-view generation and reconstruction priors. At the core of our approach is the usage of a multi-view diffusion model as a strong generative prior to perform consistent drag editing over multiple rendered views, which is followed by a reconstruction model that reconstructs 3D Gaussians of the edited object. While the initial 3D Gaussians may suffer from misalignment between different views, we address this via view-specific deformation networks that adjust the position of Gaussians to be well aligned. In addition, we propose a multi-view score function that distills generative priors from multiple views to further enhance the view consistency and visual quality. Extensive experiments demonstrate that MVDrag3D provides a precise, generative, and flexible solution for 3D drag-based editing, supporting more versatile editing effects across various object categories and 3D representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。