arXiv:2605.15186cs.CVcs.AI2026-05

用文本直接编辑3D场景,速度快且保持多视角一致。

VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Prediction

论文配图:VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Prediction
图 1 · 摘自论文原文
  • 文本指令与深度同步注入,确保语义定位准确
  • 残差场预测直接生成3D形变,保留背景稳定
  • 支持实时交互,适合动态编辑场景的应用

高质量3D场景重建已迈向通用的前馈架构,可在单次前向传播中生成复杂环境。然而,尽管在静态场景感知上表现优异,这些模型对动态人类指令的响应能力仍受限,制约其在交互式应用中的使用。现有编辑方法通常采用2D升维策略,即独立编辑各视角后回传至3D空间,该间接流程常导致纹理模糊和几何不一致,因2D编辑器缺乏跨视图的空间感知能力。为此,我们提出VGGT-Edit,一种基于文本条件的原生3D场景前馈编辑框架。该方法引入深度同步的文本注入机制,将语义引导与主干网络的空间姿态对齐,实现稳定的指令定位。随后,通过残差变换头直接预测3D几何位移以形变场景,同时保持背景稳定。为保证高保真结果,采用包含多目标项的损失函数,强化几何精度与跨视图一致性。我们还构建了DeltaScene数据集,通过自动化流水线生成并经3D一致性过滤,确保真实标签质量。实验表明,VGGT-Edit显著优于2D升维基线,在物体细节清晰度、多视角一致性方面表现更优,且具备近实时推理速度。

原文摘要 · Abstract (English)

High-quality 3D scene reconstruction has recently advanced toward generalizable feed-forward architectures, enabling the generation of complex environments in a single forward pass. However, despite their strong performance in static scene perception, these models remain limited in responding to dynamic human instructions, which restricts their use in interactive applications. Existing editing methods typically rely on a 2D-lifting strategy, where individual views are edited independently and then lifted back into 3D space. This indirect pipeline often leads to blurry textures and inconsistent geometry, as 2D editors lack the spatial awareness required to preserve structure across viewpoints. To address these limitations, we propose VGGT-Edit, a feed-forward framework for text-conditioned native 3D scene editing. VGGT-Edit introduces depth-synchronized text injection to align semantic guidance with the backbone's spatial poses, ensuring stable instruction grounding. This semantic signal is then processed by a residual transformation head, which directly predicts 3D geometric displacements to deform the scene while preserving background stability. To ensure high-fidelity results, we supervise the framework with a multi-term objective function that enforces geometric accuracy and cross-view consistency. We also construct the DeltaScene Dataset, a large-scale dataset generated through an automated pipeline with 3D agreement filtering to ensure ground-truth quality. Experiments show that VGGT-Edit substantially outperforms 2D-lifting baselines, producing sharper object details, stronger multi-view consistency, and near-instant inference speed. The project page is https://chriszkxxx.github.io/VGGT-Edit/.

3D编辑文本控制前馈模型几何形变

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。