无需训练即可编辑3D场景,用网格替代NeRF实现插入、替换、删除操作。
Towards a Training Free Approach for 3D Scene Editing
- 用网格表示替代NeRF,实现无需训练的3D编辑
- 支持文本驱动的插入、替换、删除,可组合完成复杂编辑
- 提出新算法自动定位物体放置位置,适合实时交互应用
文本驱动的扩散模型在图像编辑中表现卓越,但在3D场景编辑中,现有方法多依赖为每个场景训练NeRF。近期方法利用2D扩散模型生成编辑并投影到3D空间,但需强位置先验与文本提示以定位编辑区域,仅适用于小型、特定场景,且每次编辑需重新训练,无法实现实时编辑。为此,本文提出FreeEdit,一种基于网格表示的无训练3D场景编辑方法。得益于基础模型的发展,该方法可直接使用文本提示识别应插入、替换或删除的对象及其位置。我们定义插入、替换、删除为基本操作,通过组合实现复杂编辑。同时引入新算法自动确定物体在目标对象上的最优放置位置。在多种场景上通过定量与定性评估对比基线模型,验证了方法的有效性与通用性。
原文摘要 · Abstract (English)
Text driven diffusion models have shown remarkable capabilities in editing images. However, when editing 3D scenes, existing works mostly rely on training a NeRF for 3D editing. Recent NeRF editing methods leverages edit operations by deploying 2D diffusion models and project these edits into 3D space. They require strong positional priors alongside text prompt to identify the edit location. These methods are operational on small 3D scenes and are more generalized to particular scene. They require training for each specific edit and cannot be exploited in real-time edits. To address these limitations, we propose a novel method, FreeEdit, to make edits in training free manner using mesh representations as a substitute for NeRF. Training-free methods are now a possibility because of the advances in foundation model's space. We leverage these models to bring a training-free alternative and introduce solutions for insertion, replacement and deletion. We consider insertion, replacement and deletion as basic blocks for performing intricate edits with certain combinations of these operations. Given a text prompt and a 3D scene, our model is capable of identifying what object should be inserted/replaced or deleted and location where edit should be performed. We also introduce a novel algorithm as part of FreeEdit to find the optimal location on grounding object for placement. We evaluate our model by comparing it with baseline models on a wide range of scenes using quantitative and qualitative metrics and showcase the merits of our method with respect to others.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。