用生成式AI实现沉浸式3D场景实时编辑，支持语言与直接操作
Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs
- 模块化架构整合生成式AI，支持多模态控制输入
- 引入代理表示，在高延迟操作中保持交互流畅性
- 实验证明非文本输入更激发创作灵感，适合创意设计者
3D场景创作是空间计算应用的核心任务。现有方法分为两类：一是沉浸式、直接操纵3D内容，二是利用生成式AI（如NeRFs、3D Gaussian Splatting）捕捉真实场景并进行高层次抽象修改，但后者存在高延迟问题。本文提出Dreamcrafter，一个基于VR的3D Radiance Field实时编辑系统，具备：(1) 模块化架构，可灵活集成生成式AI算法；(2) 多层级控制方式，支持自然语言与直接操纵；(3) 代理表示机制，保障在高延迟操作中的交互体验。我们通过实证研究发现用户对控制方式的偏好，并探讨了超越文本输入的生成式界面如何提升场景构建中的创造力。
原文摘要 · Abstract (English)
Authoring 3D scenes is a central task for spatial computing applications. Competing visions for lowering existing barriers are (1) focus on immersive, direct manipulation of 3D content or (2) leverage AI techniques that capture real scenes (3D Radiance Fields such as, NeRFs, 3D Gaussian Splatting) and modify them at a higher level of abstraction, at the cost of high latency. We unify the complementary strengths of these approaches and investigate how to integrate generative AI advances into real-time, immersive 3D Radiance Field editing. We introduce Dreamcrafter, a VR-based 3D scene editing system that: (1) provides a modular architecture to integrate generative AI algorithms; (2) combines different levels of control for creating objects, including natural language and direct manipulation; and (3) introduces proxy representations that support interaction during high-latency operations. We contribute empirical findings on control preferences and discuss how generative AI interfaces beyond text input enhance creativity in scene editing and world building.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。