无需成对数据,用大模型知识蒸馏实现高效3D编辑。
Learning 3D Editing without Paired Supervision via Generative Prior Distillation

- 从图像和视觉语言模型中蒸馏视觉、语义与几何先验知识。
- 在多个视角下保持指令遵循与原始形态一致性,避免结构漂移。
- 适合希望快速生成高质量3D内容的设计师与开发者使用。
指令引导的3D编辑对交互式内容创作至关重要,但受限于高质量成对训练数据的稀缺。现有方法或依赖缓慢的测试时优化,或使用复杂流程构建伪配对数据,常引发结构偏移与几何伪影。本文提出一种无需3D配对监督的前馈式3D编辑框架——生成先验蒸馏(Generative Prior Distillation)。核心思想是通过可微渲染管道,将强大基础模型中的视觉、语义与几何知识直接蒸馏至3D编辑模型。具体地,利用图像编辑模型在主编辑视图提供2D视觉先验,通过视觉-语言模型在新视图提供语义先验,确保严格遵循指令并保留源身份。为解决2D投影监督带来的几何坍缩与多视图不一致问题,引入3D感知分布匹配正则项,该正则在3D隐空间操作,约束编辑输出位于预训练图像到3D教师模型定义的真实3D资产流形内。大量实验表明,本方法在指令忠实度与跨视图一致性上显著优于现有最优基线。
原文摘要 · Abstract (English)
Instruction-guided 3D editing is essential for interactive content creation, yet it faces a significant bottleneck: the severe scarcity of high-quality paired training data. Existing approaches attempt to bypass this by either relying on slow test-time optimization or training on pseudo-pairs constructed via complex pipelines, which often introduce structural drift and geometric artifacts. In this paper, we propose a novel framework that learns feed-forward 3D editing without paired 3D supervision via Generative Prior Distillation. Instead of relying on ground-truth 3D pairs, our core idea is to distill visual, semantic, and geometric knowledge from powerful foundation models directly into a 3D editing model. Specifically, through a differentiable rendering pipeline, we supervise the 3D representation using two complementary signals: a 2D visual prior from an image editing model at the main editing view, and a semantic prior from a Vision-Language Model at novel views to ensure strict instruction following and source identity preservation. Crucially, to address the geometric collapse and multi-view inconsistencies inherent in 2D projection supervision, we introduce a 3D-aware Distribution Matching regularization. Acting as a geometric prior, this term operates in the 3D latent space, constraining the edited output to remain within the manifold of realistic 3D assets defined by a pretrained image to 3D teacher model. Extensive experiments demonstrate that our method achieves superior instruction fidelity and cross-view consistency, significantly outperforming state-of-the-art baselines. Our project is available at: https://github.com/thiamine128/PriorEdit3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。