用强化学习让2D编辑结果在3D视角下保持一致
Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editing
- 用3D基础模型VGGT生成奖励信号，指导2D编辑对齐3D一致性
- 在无成对训练数据情况下，实现稳定多视角一致的编辑效果
- 适合需要高质量3D一致编辑的应用场景，如虚拟内容生成
利用2D扩散模型的先验知识进行3D编辑已成为一种有前景的方法。然而，编辑结果的多视角一致性仍具挑战性，且缺乏成对的3D一致编辑数据，使得监督微调（SFT）虽有效但难以实施。本文观察到：虽然生成多视角一致的3D内容非常困难，但验证3D一致性是可行的，这自然使强化学习（RL）成为可行方案。受此启发，我们提出RL3DEdit，一个由强化学习优化驱动的单阶段框架，其奖励信号源自3D基础模型VGGT。具体而言，我们利用VGGT从海量真实世界数据中学习到的强先验，将编辑后的图像输入其中，通过输出的置信度图和姿态估计误差构建奖励信号，从而通过强化学习将2D编辑先验锚定在3D一致流形上。大量实验表明，RL3DEdit在编辑质量与多视角一致性方面均优于现有最先进方法，且效率较高。为推动3D编辑发展，我们将开源代码与模型。
原文摘要 · Abstract (English)
Leveraging the priors of 2D diffusion models for 3D editing has emerged as a promising paradigm. However, multi-view consistency remains challenging in edited results, and the extreme scarcity of paired 3D-consistent editing data makes supervised fine-tuning (SFT) impractical, despite its effectiveness for editing tasks. In this paper, we observe that, while generating multi-view consistent 3D content is highly challenging, verifying 3D consistency is tractable, naturally positioning reinforcement learning (RL) as a feasible solution. Motivated by this, we propose RL3DEdit, a single-pass framework driven by RL optimization with novel rewards derived from the 3D foundation model, VGGT. Specifically, we leverage VGGT's robust priors learned from massive real-world data, feed the edited images into it, and utilize the output confidence maps and pose estimation errors as reward signals, effectively anchoring the 2D editing priors onto a 3D-consistent manifold via RL. Extensive experiments demonstrate that RL3DEdit achieves stable multi-view consistency and outperforms state-of-the-art methods in editing quality with high efficiency. To promote the development of 3D editing, we will release the code and model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。